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Abstract 


There  has  been  a  great  deal  of  discussion  of  late  about  what  it  takes  for  organizations  to  attain  high  ma¬ 
turity  status  and  what  they  can  reasonably  expect  to  gain  by  doing  so.  Clarification  is  needed  along 
with  good  examples  of  what  has  worked  well  and  what  has  not.  This  may  be  particularly  so  with  re¬ 
spect  to  measurement  and  analysis.  This  report  contains  results  from  a  survey  of  high  maturity  organi¬ 
zations  conducted  by  the  Software  Engineering  Institute  (SET)  in  2008.  The  questions  center  on  the  use 
of  process  performance  modeling  in  those  organizations  and  the  value  added  by  that  use.  The  results 
show  considerable  understanding  and  use  of  process  performance  models  among  the  organizations  sur¬ 
veyed;  however  there  is  also  wide  variation  in  the  respondents’  answers.  The  same  is  true  for  the  survey 
respondents’  judgments  about  how  useful  process  performance  models  have  been  for  their  organizations. 
As  is  tme  for  less  mature  organizations,  there  is  room  for  continuous  improvement  among  high  maturity 
organizations.  Nevertheless,  the  respondents’  judgments  about  the  value  added  by  doing  process  perform¬ 
ance  modeling  also  vary  predictably  as  a  function  of  the  understanding  and  use  of  the  models  in  their  re¬ 
spective  organizations.  More  widespread  adoption  and  improved  understanding  of  what  constitutes  a  suit¬ 
able  process  performance  model  holds  promise  to  improve  CMMI-based  performance  outcomes 
considerably. 
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1  Introduction 


There  has  been  a  great  deal  of  discussion  of  late  about  just  what  it  takes  for  organizations  to  attain  high 
maturity  status  and  what  they  can  reasonably  expect  to  gain  by  doing  so.  Clarification  is  needed  along 
with  good  examples  of  what  has  worked  well  and  what  has  not.  This  may  be  particularly  so  with  re¬ 
spect  to  measurement  and  analysis  and  the  use  of  process  performance  modeling.  By  process  perform¬ 
ance  modeling,  we  refer  to  the  use  of  analytic  methods  to  construct  process  performance  models  and 
establish  baselines. 

Such  discussion  needs  to  be  conducted  in  a  spirit  of  continuous  improvement.  Just  as  CMMI  models 
have  matured  since  the  CMM  for  Software  first  appeared,  more  and  more  organizations  have  achieved 
higher  levels  of  capability  and  maturity  in  recent  years  [Paulk  1993,  SEI  2002,  SEI  2008b].  Yet  there  is 
room  for  further  process  improvement  and  for  better  understanding  of  how  high  maturity  practices  can 
lead  to  better  project  performance  and  quality  outcomes  [Stoddard  2008]. 

This  report  contains  results  from  a  survey  of  high  maturity  organizations  conducted  by  the  Software 
Engineering  Institute  (SEI)  in  2008.  The  questions  center  on  the  use  of  process  performance  modeling 
in  those  organizations  and  the  value  added  by  that  use.  There  is  evidence  of  considerable  understanding 
and  use  of  process  performance  models  among  the  organizations  surveyed;  however  there  also  is  varia¬ 
tion  in  responses  among  the  organizations  surveyed.  The  same  is  true  for  the  survey  respondents’ 
judgments  about  how  useful  process  performance  models  have  been  for  their  organizations.  As  is  true 
for  less  mature  organizations,  room  remains  for  continuous  improvement  among  high  maturity  organi¬ 
zations.  Nevertheless,  the  respondents’  judgments  about  the  value  added  by  doing  process  performance 
modeling  also  vary  predictably  as  a  function  of  the  understanding  and  use  of  the  models  in  their  respec¬ 
tive  organizations.  More  widespread  adoption  and  improved  understanding  of  what  constitutes  a  suit¬ 
able  process  performance  model  holds  promise  to  improve  CMMI-based  performance  outcomes  con¬ 
siderably. 

Similar  results  have  been  found  in  two  other  surveys  in  the  SEI  state  of  measurement  and  analysis  prac¬ 
tice  survey  series  [Goldenson  2008a,  Goldenson  2008b].'  Based  on  organizations  across  the  full  spec¬ 
trum  of  CMMI-based  maturity  in  2007  and  2008,  both  studies  provide  evidence  about  the  circum¬ 
stances  under  which  measurement  and  analysis  capabilities  and  performance  outcomes  are  likely  to 
vary  as  a  consequence  of  achieving  higher  CMMI  maturity  levels.  Most  of  the  differences  reported  are 
consistent  with  expectations  based  on  CMMI  guidance,  which  provides  confidence  in  the  validity  of 
the  model  structure  and  content.  Instances  exist  where  considerable  room  for  organizational  improve¬ 
ment  remains,  even  at  maturity  levels  4  and  5.  However,  the  results  typically  show  characteristic,  and 
often  quite  substantial,  differences  associated  with  CMMI  maturity  level.  Although  the  distributions 
vary  somewhat  across  the  questions,  there  is  a  common  stair-step  pattern  of  improved  measurement  and 
analysis  capabilities  that  rises  along  with  reported  performance  outcomes  as  the  survey  respondents’ 
organizations  move  up  in  maturity  level. 


The  SEI  state  of  measurement  and  analysis  survey  series  began  in  2006  [Kasunic  2006].  The  focus  in  the  first  two 
years  was  on  the  use  of  measurement  and  analysis  in  the  wider  software  and  systems  engineering  community.  A 
comparable  survey  was  fielded  in  2008.  This  is  the  first  year  the  study  also  focused  on  measurement  and  analysis 
practices  in  high  maturity  organizations. 
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The  survey  described  in  this  report  is  organized  around  several  important  issues  faced  in  the  adoption 
and  use  of  measurement  and  analysis  in  high  maturity  organizations.  Confusion  still  exists  in  some 
quarters  about  the  intent  of  CMMI  with  respect  to  the  practice  of  measurement  and  analysis.  Hence 
questions  have  been  asked  to  identify  the  extent  to  which  such  practices  do  or  do  not  meet  that  intent. 
Questions  focus  particularly  on  what  constitutes  a  suitable  process  performance  model  in  a  CMMI  con¬ 
text.  Related  questions  ask  about  the  breadth  of  statistical,  experimental,  and  simulation  methods  used; 
attention  paid  to  data  quality  and  integrity;  staffing  and  resources  devoted  to  the  work;  pertinent  train¬ 
ing  and  coaching;  and  the  alignment  of  the  models  with  business  and  technical  objectives.  Equally  im¬ 
portantly,  the  survey  respondents  were  queried  about  technical  challenges  and  other  barriers  to  and  fa¬ 
cilitators  of  successful  adoption  and  use  of  process  performance  models  in  their  organizations. 

The  remainder  of  this  document  is  broken  up  into  eight  sections  and  three  appendices,  followed  by  the 
references.  A  description  of  the  survey  respondents  and  their  organizations  is  contained  in  Section  2. 
Basic  descriptions  of  variability  in  reported  process  performance  and  its  outcomes  can  be  found  in  Sec¬ 
tion  3.  Section  4  contains  similar  descriptions  of  process  performance  models  and  related  processes. 
Differences  in  organizational  sponsorship  for  and  technical  challenges  facing  process  improvement  in 
this  area  are  summarized  in  Section  5.  The  results  in  Sections  3  to  5  will  be  used  over  time  to  identify 
changes  and  trends  in  how  measurement  and  analysis  is  performed  in  high  maturity  organizations. 

The  extent  to  which  variability  in  process  outcomes  can  be  explained  by  concurrent  differences  in 
process  and  organizational  context  is  examined  in  Section  6.  Selected  results  from  the  2008  companion 
survey  based  on  organizations  from  across  the  full  spectrum  of  CMMI-based  maturity  levels  are  found 
in  Section  7.  These  results  highlight  the  potential  payoff  of  measurement  and  analysis  in  lower  maturity 
organizations.  The  report  summary  and  conclusions  are  in  Section  8.  The  questionnaires  and  invitations 
to  participate  in  the  surveys  are  reproduced  in  Appendix  A  and  Appendix  B.  Appendix  C  contains  free 
form  textual  replies  from  the  respondents  to  the  high  matnrity  snrvey,  which  we  hope  will  provide  a 
better  qualitative  sense  of  the  meaning  that  can  be  attributed  to  the  quantitative  results. 

This  survey  is  the  most  comprehensive  of  its  kind  yet  done.  We  expect  it  to  be  of  considerable  value  to 
organizations  wishing  to  continue  improving  their  measurement  and  analysis  practices.  We  hope  that 
the  results  will  allow  you  to  make  useful  comparisons  with  similar  organizations  and  provide  practical 
guidance  for  continuing  improvement  in  your  own  organization. 
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2  The  Respondents  and  Their  Organizations 


Invitations  to  participate  in  the  survey  were  sent  to  the  sponsors  of  all  organizations  recorded  by  the 
SEI  as  having  been  appraised  for  their  compliance  with  CMMI  maturity  levels  four  or  five  over  the 
five-year  period  ending  in  March  2008.  Personalized  announcements  about  the  survey,  invitations,  and 

up  to  three  reminders  to  participate  were  sent  in  May  and  June.  A  total  of  156  questionnaires  were  re- 

2 

turned  for  a  46  percent  completion  rate. 

Most  or  all  of  the  survey  respondents  were  in  a  position  to  be  conversant  with  the  use  of  process  per¬ 
formance  baselines  and  models  in  their  respective  organizations.  Including  most  of  those  who  chose  the 
“other”  category  in  Figure  1,  the  majority  of  the  survey  respondents  filled  management  roles  in  their 
organizations.  As  can  be  inferred  by  some  of  the  less  populated  categories,  some  of  the  sponsors  dele¬ 
gated  completion  of  their  questionnaires  to  others.  We  also  know  from  email  and  voice  exchanges  that 
some  sponsors  completed  them  jointly  with  knowledgeable  staff  members,  including  process  and  qual¬ 
ity  engineers.  While  almost  a  third  of  the  respondents  characterized  themselves  largely  as  users  of 
measurement-based  information,  closer  to  two-thirds  of  them  said  that  they  were  both  providers  and 
users  of  such  information  (see  Figure  2). 


As  can  be  seen  throughout  the  remainder  of  this  report,  over  90  percent  of  the  respondents  answered  most  or  aii  of 
the  survey  questions. 


3  I  CMU/SEI-2008-TR-024 


How  would  you  best  describe  your  involvement 
with  measurement  and  analysis? 


A  provider  of 
measurement-based 
information 
4% 


Other 

3% 


N  = 


156 


Figure  2:  Measurement  and  analysis  roles  of  survey  respondents 

Notice  in  Figure  3  that  three  quarters  of  the  respondents  from  the  high  maturity  organizations  reported 
that  they  were  from  CMMI  maturity  level  five  organizations.  That  is  consistent  with  the  proportions  for 
maturity  levels  four  and  five  as  reported  in  the  most  recent  Process  Maturity  Profile  [SEI  2008a]. 

A  comparison  of  their  reported  and  appraised  maturity  levels  also  suggests  that  the  respondents  were 
being  candid  in  their  replies.  The  current  maturity  levels  reported  hy  90  percent  of  the  respondents  do 
in  fact  match  the  levels  at  which  their  respective  organizations  were  most  recently  appraised.  Of  the 
remaining  ten  percent,  four  respondents  reported  higher  maturity  levels,  one  moving  from  level  three  to 
level  five  and  three  moving  from  level  four  to  level  five.  Of  course,  it  is  quite  reasonable  to  expect 
some  post-appraisal  improvement. 
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Figure  3:  CMMI  maturity  levels  of  responding  organizations 


More  interestingly  however,  twelve  respondents  reported  maturity  levels  that  were  lower  than  their  ap¬ 
praised  maturity  levels.  One  of  the  28  whose  organizations  were  appraised  at  maturity  level  four  reported 
being  at  level  three,  and  eleven  of  the  124  appraised  at  maturity  level  five  reported  no  longer  being  at  that 
level.  One  of  them  reported  currently  being  at  maturity  level  four,  another  person  answered  “don’t  know” 
and  nine  reported  that  they  were  now  at  maturity  level  three.  These  individuals  apparently  were  reporting 
about  perceived  changes  in  their  organizational  units  or  perhaps  perceived  changes  in  appraisal  criteria 
that  led  to  a  lower  process  maturity  level  in  their  view. 


The  remaining  charts  in  this  section  characterize  the  mix  of  organizational  units  that  participated  in  the 
survey.  As  noted  earlier,  we  intend  to  track  these  or  similar  factors  over  time,  along  with  changes  in  the 
use  and  reported  value  of  measurement  and  analysis  in  CMMI-based  process  improvement. 


More  than  a  quarter  of  the  organizational  units  are  government  contractors  or  government  organizations 
(see  Figure  4).  About  two-thirds  of  them  focus  their  work  on  product  or  system  development  (see 
Figure  5).  Almost  all  of  them  focus  their  work  on  software  engineering,  although  large  proportions  also 
report  working  in  other  engineering  disciplines  (see  Figure  6).  The  numbers  of  their  full-time  employ¬ 
ees  who  work  predominately  in  software,  hardware,  or  systems  engineering  are  distributed  fairly  evenly 
from  200  or  fewer  through  more  than  2000  (see  Figure  7).  Over  half  of  the  participating  organizations 
are  primarily  located  in  the  United  States  or  India;  no  other  country  exceeds  four  percent  of  the  re¬ 
sponding  organizations,  with  the  exception  of  China  which  accounts  for  15  percent  (see  Figure  8). 


The  “other”  category  contains  various  combinations  of  the  rest  of  the  categories,  inciuding  defense  but  aiso  iT,  main¬ 
tenance,  system  integration  and  service  provision. 
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How  is  your  organization  best  described? 


Other  government 
agency 

1% 

Department  of 
Defense  or  military 
organization 
2% 

Other  government 
contractor 
5% 


Defense  contractor 
20% 


Other 

19% 


Commercial  off  the 
shelf 
5% 


Contracted  new 
development 
32% 


In-house  or 
proprietary 
development  or 
maintenance 
16% 


N  =  155 


Figure  4:  Sectors  represented  in  responding  organizations 


What  is  the  primary  focus  of  your  organization’s  work? 


Other 

10% 


Figure  5:  Product  and  service  focus  of  responding  organizations 
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What  kinds  of  engineering  are  major 
parts  of  your  organization’s  work? 


engineering  engineering  engineering  engineering  engineering 

N  =  156 


Figure  6:  Mix  of  engineering  activities  in  responding  organizations 


Approximately  how  many  full-time  employees  in  your  organization  work 
predominantly  in  software,  hardware  or  systems  engineering? 


Figure  7:  Number  of  fuli-time  engineering  employees  in  responding  organizations 
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In  what  country  is  your  organization  primarily  located? 


29% 


Figure  8:  Primary  iocation  of  responding  organizations 


N  =  155 
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3  Baselining  Variability  in  Process  Outcomes:  Value  Added  by 
Process  Performance  Modeling 


There  currently  is  considerable  interest  in  process  performance  models  among  organizations  that  have 
achieved  or  aspire  to  CMMI  high  maturity  status.  The  development  of  such  models  is  discussed  in  the 
Organizational  Process  Performance  (OPP)  process  area,  and  the  varied  uses  of  process  performance 
models  is  covered  in  the  Quantitative  Project  Management  (QPM),  Causal  Analysis  and  Resolution 
(CAR),  and  Organizational  Innovation  and  Deployment  (OID)  process  areas.  However,  further  clarifi¬ 
cation  is  needed  along  with  good  examples  of  what  has  worked  well  and  what  has  not.  Questions  about 
process  performance  models  form  a  predominant  theme  in  the  snrvey  to  address  this  sitnation. 


The  survey  respondents’  reports  about  the  effects  of  their  process  performance  modeling  efforts  are 
described  in  this  section.  They  were  first  asked  a  series  of  questions  describing  several  specific  effects 
that  have  been  experienced  in  a  number  of  high  maturity  organizations  [Goldenson  2007a,  Gibson 
2006,  Goldenson  2003b,  Stoddard  2008].  As  shown  in  Figure  9,  over  three-quarters  of  them  reported 
experiencing  better  project  performance  or  product  quality  frequently  or  almost  always.  A  similar  inci¬ 
dence  of  fewer  project  failures  was  reported  by  over  60  percent  of  the  respondents.  Almost  as  many 
respondents  reported  a  similar  incidence  of  better  tactical  decision  making,  and  over  40  percent  said 
that  they  had  experienced  comparable  improvements  in  strategic  decision  making  as  a  result  of  their 
process  performance  modeling  activities. 


Following  are  a  few  statements  about  the  possible  effects  of 
using  process  performance  modeling.  To  what  extent  do 
they  describe  what  your  organization  has  experienced? 
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□  Not  applicable 

□  Don't  know 
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□  About  half  the  time 

■  Frequently 

□  Almost  always 
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Figure  9:  Effects  attributed  to  using  process  performance  modeling 
These  results  add  insight  about  the  benefits  of  process  performance  models;  however  the  question 
series  also  served  to  get  the  respondents  thinking  about  how  they  could  best  characterize  the  over- 
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all  value  of  process  performance  modeling  for  their  organizations.  The  respondents’  answers  to  a 
single  question  about  such  an  effect  are  shown  in  Figure  10.  As  the  figure  shows,  over  half  of  the 
respondents  said  that  they  have  obtained  much  useful  information  from  their  process  performance 
models,  which  proved  to  be  very  valuable  to  their  organizations.  Another  eight  percent  said  that 
the  models  have  been  extremely  valuable  and  that  they  could  not  do  their  work  properly  without 
them.  While  almost  40  percent  said  that  their  process  performance  efforts  have  yielded  only 
mixed  value,  only  a  handful  reported  gaining  little  or  no  value  from  their  models,  and  none  of  the 
survey  respondents  chose  the  option  that  their  modeling  efforts  had  been  harmful  overall. 


Overall,  how  useful  have  process  performance 
models  been  for  your  organization? 


It’s  been  harmful,  not 
helpful 
0% 

Don't  know 
0% 


Little  or  no  value 
2% 


Mixed  value  --  we 
have  obtained  useful 
information  on 
occasion 
38% 


Extremely  valuable  -- 
we  couldn’t  do  our 
work  properly  without 
them 
8% 


Very  valuable  --  we 
have  obtained  much 
useful  information 
from  them 
52% 


N  =  144 


Figure  1 0:  Overall  effect  attributed  to  process  performance  modeling 

As  noted  in  Section  1,  the  responses  to  the  questions  on  specific  and  overall  benefits  of  process  per¬ 
formance  models  are  intended  to  serve  as  barometers  of  community  adoption  to  be  tracked  over  the 
next  several  years.  They  also  are  crucial  for  explaining  variability  in  outcomes  of  process  performance 
modeling.  We  have  relied  in  Section  6  on  the  question  about  the  overall  effect  attributed  by  the  survey 
respondents  to  their  organizations’  modeling  efforts.  The  questions  about  specific  effects  of  the  model¬ 
ing  are  not  necessarily  equally  pertinent  to  all  of  the  surveyed  organizations;  however,  similar  results  to 
those  based  on  the  overall  question  also  exist  using  a  composite  measure  of  the  specific  effects.'* 


Note  that  some  of  the  questions  described  In  Sections  4  and  5  that  are  used  as  x  variables  In  Section  6  also  have 
been  modeled  or  discussed  there  as  Interim  y  variables. 
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4  Baselining  Variability  in  Process  Implementation 


The  survey  respondents’  reports  about  the  extent  to  which  their  process  activities  differed  are  described 
here.  These  include  the  respondents’  reports  about  stakeholder  involvement  in  setting  measurement  and 
analysis  goals  and  objectives;  measurement-related  training;  their  understanding  and  use  of  process 
performance  models;  and  their  use  of  various  data  analytic  methods.  As  described  in  more  detail  in 
Section  6,  several  composite  variables  based  on  related  groups  of  these  questions  and  individual  ques¬ 
tions  described  here  vary  predictably  with  the  effects  attributed  by  the  survey  respondents  to  their  or¬ 
ganizations’  modeling  efforts. 

4.1  Stakeholder  Involvement  in  Setting  Measurement  and  Analysis  Goals  and  Objectives 

The  importance  of  stakeholder  involvement  for  process  improvement  in  general  and  measurement  and 
analysis  in  particular  is  widely  acknowledged.  This  can  be  seen  in  CMMI  generic  practice  2.7  which 
emphasizes  the  importance  of  identifying  and  involving  relevant  stakeholders  during  process  execution. 
Such  notions  are  basic  to  goal-driven  measurement  [Park  1996,  van  Solingen  1999].  It  also  is  crucial 
for  the  CMMI  Measurement  and  Analysis  process  area,  particularly  in  specific  goal  1  and  specific  prac¬ 
tice  1.1,  which  are  meant  to  ensure  that  measurement  objectives  and  activities  are  aligned  with  the  or¬ 
ganizational  unit’s  information  needs  and  objectives,  and  specific  practice  2.4,  which  emphasizes  the 
importance  of  reporting  the  results  to  all  relevant  stakeholders.  Empirical  evidence  indicates  that  the 
existence  of  such  support  increases  the  likelihood  of  the  snccess  of  measurement  programs  in  general 
[Goldenson  1999,  Goldenson  2000,  Gopal  2002]. 

Experience  with  several  leading  high  maturity  clients  has  shown  that  insufficient  stakeholder  involve¬ 
ment  in  the  creation  of  process  performance  baselines  and  models  can  seriously  jeopardize  the  likeli¬ 
hood  of  using  them  productively.  Stakeholders  must  actively  participate  not  only  in  what  outcomes  are 
worthy  of  being  predicted  with  the  models,  bnt  also  in  identifying  the  controllable  x  factors  that  have 
the  greatest  potential  to  assist  in  predicting  those  outcomes.  Many  organizations  pursning  baselines  and 
models  make  the  mistake  of  identifying  the  outcomes  and  controllable  factors  from  the  limited  perspec¬ 
tive  of  the  quality  and  process  improvement  teams. 

We  asked  a  series  of  questions  seeking  feedback  on  the  involvement  of  eight  different  categories  of 
potential  stakeholders.  As  shown  in  Eigure  11,  these  groups  are  commonly  present  in  most  organiza¬ 
tional  and  project  environments.  The  groups  are  listed  in  Pareto  order  by  the  proportions  of  respondents 
who  reported  that  those  stakeholder  groups  were  either  substantially  or  extensively  involved  in  decid¬ 
ing  on  plans  of  action  for  measurement  and  analysis  in  their  organizations.  Notice  that  process/quality 
engineers  and  measurement  specialists  are  more  likely  to  have  the  most  involvement,  while  project  en¬ 
gineers  and  customers  have  the  least. 

Observations  by  SEI  technical  staff  and  others  suggest  that  successful  high  maturity  organizations  in¬ 
volve  stakeholders  from  a  variety  of  disciplines  in  addition  to  those  summarized  in  Eigure  11,  such  as 
marketing,  systems  engineering,  architecture,  design,  programming,  test,  and  snpply  chain.  This  ques¬ 
tion  will  also  be  profiled  across  time  to  gauge  the  impact  of  richer  involvement  of  other  stakeholder 
groups. 
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How  would  you  characterize  the  involvement  of  various  potential  stakeholders  in 
setting  goals  and  deciding  on  plans  of  action  for  measurement  and  analysis  in 

your  organization? 


□  Don't  know 

□  Does  not  apply 
■  Little  if  any 

□  Limited 

□  Moderate 

□  Substantial 

□  Extensive 


Figure  11:  Stakeholder  involvement  in  responding  organizations 


4.2  Measurement-Related  Training 

The  questions  in  this  section  aim  to  provide  insight  into  the  type  and  volume  of  training  done  by  the 
surveyed  organizations  for  the  key  job  roles  related  to  the  practice  and  use  of  process  performance 
modeling.  A  series  of  questions  also  asks  about  the  sources  of  training  and  training  materials  used  by 
these  organizations.^ 

Notice  in  Figure  12  that  the  bulk  of  training-related  activity  for  executive  and  senior  managers  was  lim¬ 
ited  to  briefings,  although  about  15  percent  of  them  have  completed  in-service  training  courses.  Middle 
and  project-level  managers  were  much  more  likely  to  take  one-  or  two-day  courses,  and  almost  twenty 
percent  of  the  project  managers  typically  took  one-  to  four-week  courses.  The  process  and  quality 
engineers  were  most  likely  to  have  taken  the  most  required  course  work.  Of  course,  some  of  these  same 
people  build  and  maintain  the  organizations’  process  performance  models.  Similar  results  can  be  seen 
in  Figure  13  for  those  who  build,  maintain,  and  use  the  organizations’  process  performance  models. 
Figure  14  summarizes  the  sources  by  which  the  surveyed  organizations  ensured  that  their  process 
performance  model  builders  and  maintainers  were  properly  trained.  Almost  90  percent  relied  on 
training  developed  and  delivered  within  their  own  organizations.  However,  over  60  percent  contracted 
external  training  services,  and  almost  30  percent  purchased  training  materials  developed  elsewhere  to 
use  in  their  own  courses.  About  a  third  of  them  emphasized  that  they  hire  people  based  on  their 
existing  expertise. 


SEI  experience  regarding  the  nature  and  degree  of  training  and  their  associated  outcomes  of  process  performance 
modeling  suggest  that  5  to  1 0  percent  of  an  organization’s  headcount  must  be  trained  In  measurement  practices  and 
process  performance  modeling  techniques  to  have  a  serious  beneficial  Impact  on  the  business. 
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What  best  characterizes  the  measurement  related  training 
that  your  organization  requires  for  its  employees? 
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□  Does  not  apply 

■  Don't  know 

□  College  courses 

□  1-4  week  courses 

□  1-2  day  courses 

■  Tutorials 

□  Briefings 

□  Online/self  paced 

■  On  the  job 

□  None 
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Middle 
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or  product  line 
(N  =  149) 


Project  Project 

managers  engineers  and 
(N  =  149)  other  technical 
staff 

(N  =  149) 


Process  or 
quality 
engineers 
(N  =  149) 


Figure  12:  Measurement  related  training  required  for  employees  of  responding  organizations 


What  kind  of  specialized  training,  if  any,  does  your  organization  require  for  its 
employees  who  have  responsibilities  for  process  performance  modeling? 
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Figure  13:  Specialized  training  on  process  performance  modeiing  in  responding  organizations 
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In  what  ways  does  your  organization  ensure  that  its  process  performance 
modei  buiiders  and  maintainers  are  properiy  trained? 
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Contracts  with  Conferences  & 

We  hire  the 

Purchase  of 

Other 

developed  & 
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right  people  in 
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Figure  14:  Sources  of  process  performance  model  expertise  in  responding  organizations 

Earlier  observations  by  SEI  technical  staff  and  others  underscore  the  importance  of  measurement  train¬ 
ing  and  coaching  for  management  in  addition  to  the  training  for  measurement  analysts  and  model 
builders.  Several  of  the  questions  used  in  this  survey  and  other  related  questions  will  be  profiled  over 
time  to  help  gauge  the  state  of  the  practice  in  training  on  measurement  and  analysis.  The  results  will  be 
used  to  help  guide  future  training  and  services  offered  by  the  SEI.  Training  remains  one  of  the  most 
important  and  problematic  measurement  deployment  issues  within  many  organizations.  Many  organi¬ 
zations  believe  they  have  fully  deployed  Six  Sigma  when  in  fact  their  staff  members  are  not  armed  with 
a  balanced  and  complete  statistical  toolkit  for  conducting  regression  analysis,  probabilistic  modeling,  or 
various  types  of  simulation  modeling. 

4.3  Process  Performance  Modeling 

Over  the  past  two  years,  SEI  technical  staff  and  others  have  worked  to  clarify  what  have  been  called  the 
“healthy  ingredients”  of  process  performance  models  [Young  2008a,  Young  2008b].  The  healthy  in¬ 
gredients  include 

•  modeling  uncertainty  in  the  model’s  predictive  factors 

•  ensuring  that  the  model  has  controllable  factors  in  addition  to  possible  uncontrollable  factors 

•  identifying  factors  to  construct  models  that  are  directly  associated  with  sub-processes 

•  predicting  final  and  interim  project  outcomes 

•  using  confidence  intervals  to  provide  a  range  of  expected  outcome  behaviors  enabling  “what-if  ’ 
analysis  using  the  model 

•  enabling  projects  to  identify  and  implement  mid-course  corrections  to  help  ensure  project  success 
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Two  series  of  questions  focused  on  the  detailed  aspects  of  these  ingredients.  Composite  measures  based 
on  answers  to  both  questions  showed  a  strong  association  with  respondents’  reports  about  the  value  of 
using  process  performance  models. 


The  first  set  of  questions  focused  on  the  emphasis  the  responding  organizations  placed  on  the  various 
healthy  ingredients  in  their  modeling  efforts.  The  results  in  Figure  15  show  that  the  organizations  var¬ 
ied  considerably  in  their  emphasis  on  these  factors.  While  many  of  the  organizations  addressed  these 
factors  quite  extensively,  room  for  improvement  exists  among  some  of  the  organizations,  particularly 
on  the  right  side  of  the  Pareto  ordering.  The  same  is  so  for  the  question  set  that  focused  on  the  purposes 
for  which  the  organizations  used  their  models.  As  seen  in  Figure  16,  there  is  considerable  room  for  im¬ 
provement  in  modeling  variation  in  process  outcomes  and  in  enabling  “what-if  ’  analyses. 


How  much  emphasis  does  your  organization  place  upon 
the  following  in  its  process  performance  modeling? 
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Figure  15:  Emphasis  on  heaithy  process  performance  model  ingredients 
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To  what  degree  are  your  organization’s  process 
performance  modeis  used  for  the  foiiowing  purposes? 


□  Does  not  apply 

□  Don't  know 

■  Little  if  any 

□  Limited 

□  Mederate 

■  Substantial 

□  Extensive 


Figure  16:  Use  of  healthy  process  performance  model  ingredients 

As  just  shown,  many  CMMI  high  maturity  organizations  do  recognize  and  implement  these  ingredients 
to  maximize  the  business  value  of  their  process  performance  modeling.  However,  better  understanding 
and  implementation  of  these  healthy  ingredients  remains  a  need  in  the  wider  community.  These  ques¬ 
tions  will  be  profiled  in  successive  surveys  for  the  next  several  years. 


SEI  technical  staff  and  others  have  observed  a  disparity  of  application  and  use  of  process  performance 
models  and  baselines  among  various  clients.  Far  greater  business  value  appears  to  accme  in  organiza¬ 
tions  where  the  use  of  models  and  baselines  includes  the  complete  business  and  its  functions  rather  than 
isolated  quality  or  process  improvement  teams.  We  asked  a  series  of  questions  to  provide  insight  into 
the  scope  of  use  of  the  process  performance  models  and  baselines  implemented  in  the  respondents’  or¬ 
ganizations.  These  include; 


•  operational  purposes  for  which  the  models  and  baselines  are  used 

•  functional  areas  where  process  performance  models  and  baselines  are  actively  used 

•  quality,  project,  and  process  performance  outcomes  to  be  predicted 

•  processes  and  activities  that  are  routinely  modeled 

As  shown  in  Figure  17,  project  monitoring,  identifying  opportunities  for  corrective  action,  project 
planning,  and  identifying  improvement  opportunities  were  the  most  common  operational  uses  noted  by 
the  survey  respondents.  All  of  these  operational  uses  were  identified  by  over  half  of  the  respondents. 
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For  what  operational  purposes  are  models  and  baselines 
routinely  used  in  your  project  and  organizational  product 
development,  maintenance  or  acquisition  activities? 
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Figure  17:  Operational  purposes  of  process  performance  modeling 


However,  the  same  cannot  be  said  for  the  functional  areas  of  the  organizations’  businesses  that  are  not 
directly  related  to  software  and  software  intensive  systems  per  se.  As  shown  in  Figure  18,  only  one  of 
these  functional  areas  was  routinely  modeled  by  more  than  50  percent  of  the  responding  organizations. 


Like  all  of  the  other  questions,  the  questions  about  operational  uses  and  other  business  functional  areas 
will  be  profiled  across  time  to  enable  an  empirical  link  to  overall  business  benefit  and  to  observe  the 
changing  profile  within  the  community  as  adoption  accelerates.  However,  these  questions  have  not 
been  used  in  our  statistical  modeling  exercises  presented  in  Section  6.  It  does  not  make  sense  to  use 
every  question  to  model  the  effects  of  process  performance  modeling.  There  is  no  theoretical  reason  to 
expect  association  for  some  of  the  questions.  Others,  such  as  these  two  question  sets,  are  not  well  dis¬ 
tributed  empirically  to  serve  as  useful  predictors. 
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Where  else  in  the  organization  are  process 
performance  models  and  baselines  used? 
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Figure  18:  Other  management  uses  of  process  performance  modeling 


The  diversity  of  process  performance  models  used  to  predict  quality,  project,  and  process  performance 
outcomes  are  another  matter  entirely.  As  can  he  seen  in  Figure  19  and  Figure  20,  there  is  considerably 
more  variation  in  the  extent  to  which  various  outcomes  and  interim  outcomes  are  predicted  by  the 
process  performance  models  used  in  the  organizations  that  participated  in  this  survey.  Delivered  de¬ 
fects,  cost,  and  schedule  duration  were  commonly  modeled  by  over  80  percent  of  the  organizations. 
Accuracy  of  estimates  was  modeled  by  close  to  three-quarters  of  them.  Estimates  at  completion  and 
escaped  defects  were  modeled  by  over  60  percent  of  the  respondents’  organizations.  However,  the 
other  outcomes  were  predicted  less  commonly. 


As  shown  in  Section  6,  a  composite  variable  based  on  the  diversity  of  models  used  to  predict  product 
quality  and  project  performance  outcomes  is  quite  strongly  related  to  the  differences  in  overall  value 
that  the  survey  respondents  attribute  to  their  organizations’  process  performance  modeling  activities.  A 
composite  variable  based  on  the  diversity  of  models  used  to  predict  interim  performance  outcomes  also 
is  related  to  the  reported  value  of  their  respective  modeling  efforts,  although  less  strongly. 
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Which  of  the  following  product  qualify  and  project  performance  outcomes  are 
routinely  predicted  with  process  performance  models  in  your  organization? 


Figure  1 9:  Diversity  of  process  performance  models:  product  quality  and  project  performance 


Which  of  the  following  (often  interim)  process  performance  outcomes  are 
routinely  predicted  with  process  performance  models  in  your  organization? 


Figure  20:  Diversity  of  process  performance  models:  process  performance 

The  processes  and  related  activities  routinely  modeled  are  summarized  in  Figure  21.  Not  surprisingly, 
project  planning,  estimation,  quality  control,  software  design,  and  coding  were  modeled  most  com- 
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monly.  Modeling  of  requirements  engineering  activities  was  reported  by  almost  half  of  the  responding 
organizations;  however,  there  is  considerable  room  for  improvement  elsewhere. 


Which  of  the  following  processes  and  activities 
are  routinely  modeled  in  your  organization? 


Figure  21:  Routinely  modeled  processes  and  activities 

Finally,  notice  in  Figure  22  that  almost  60  percent  of  the  survey  respondents  reported  that  process  per¬ 
formance  model  predictions  were  used  at  least  frequently  in  their  organizations’  status  and  milestone 
reviews.  Another  large  group  (close  to  20  percent)  reported  that  they  use  such  information  about  half 
the  time  in  their  reviews,  while  21  percent  say  they  do  so  only  occasionally  at  best. 

In  fact,  this  question  turns  out  to  be  more  closely  related  statistically  to  the  performance  outcomes  of 
doing  process  performance  modeling  of  all  the  x  variables  examined  in  Section  6.®  Some  organizations 
develop  process  performance  models  that  end  up  on  a  shelf  unused,  because  either  the  models  are  pre¬ 
dicting  irrelevant  or  insignificant  outcomes  or  they  lack  controllable  x  factors  that  provide  management 
direct  insight  on  the  action  to  take  when  predicted  outcomes  are  undesirable.  To  have  an  institutional 
effect,  process  performance  models  must  provide  breadth  and  depth  so  that  they  can  play  a  role  in  the 
various  management  reviews  within  an  organization  and  its  projects. 


This  question  also  is  of  interest  in  its  own  right  as  a  measure  of  the  value  added  by  doing  process  performance  model¬ 
ing.  Much  like  the  respondents’  direct  reports  in  Section  3  about  the  outcomes  of  doing  process  performance  model¬ 
ing,  frequency  of  use  of  the  model  predictions  for  decision  making  does  vary  as  a  function  of  the  way  in  which  the 
models  are  built  and  used,  as  well  as  the  respondents'  answers  to  other  questions  described  elsewhere  in  Sections  3 
and  4  about  differences  in  other  process  activities  and  organizational  context. 
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How  often  are  process  performance  model  predictions 
used  to  inform  decision  making  in  your  organization’s  status 
and  milestone  reviews? 

Don't  know 
1% 

Rarely  if  ever_ 


5%  I  Almost  always 


Figure  22:  Use  of  process  performance  model  predictions  in  reviews 


4.4  Analytic  Methods 

This  set  of  questions  solicited  detailed  feedback  on  the  extent  to  which  the  responding  organizations 
used  a  series  of  statistical,  probabilistic,  and  simulation  techniques.  Along  with  broad  stakeholder  in¬ 
volvement  and  management  support,  the  use  of  a  rich  variety  of  analytic  methods  has  been  shown  to  be 
closely  related  to  the  likelihood  of  the  success  of  measurement  programs  in  general  [Goldenson  1999, 
Goldenson  2000,  Gopal  2002].  The  questions  also  closely  mirror  the  analytic  methods  taught  in  SEI 
courses  as  part  of  a  toolkit  focusing  on  CMMI  high  maturity  and  process  performance  modeling.^ 

These  questions  serve  a  two-fold  purpose,  namely  to  track  the  profile  of  technique  adoption  by  the 
community  across  time  and  to  correlate  business  impact  to  the  degree  of  sophistication  of  the  analytical 
toolkits  employed.  Additionally,  the  questions  themselves  were  intended  to  introduce  the  survey  re¬ 
spondents  who  were  not  yet  familiar  with  them  to  a  full  range  of  techniques  that  might  be  used  profita¬ 
bly  in  their  own  process  performance  modeling. 

Note  in  Figure  23  that  there  is  a  wide  range  in  the  extent  to  which  the  organizations  use  these  methods. 
Various  statistical  process  control  (SPC)  methods  are  used  most  often.  Least  squares  (continuous)  re¬ 
gression  and  even  analysis  of  variance  methods  are  used  relatively  extensively.  However,  categorical 
(e.g.,  logistic  or  loglinear)  regression  and  especially  designed  experiments  are  used  much  less 


More  Information  about  these  courses,  Improving  Process  Performance  Using  Six  Sigma  and  Designing  Products  and 
Processes  Using  Six  Sigma,  can  be  found  at  http://www.sel.cmu.edu/products/courses/p49b.html  and 
http://www.sel.cmu.edu/products/courses/p56b.html. 
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extensively.  Such  methods  can  he  very  appropriate  for  categorical  and  ordered  data,  which  are  quite 
common  in  management  environments  and  software  and  systems  engineering  environments. 


As  seen  in  Section  6,  the  extent  of  use  of  these  methods  is  strongly  related  statistically  to  the  value  at¬ 
tributed  by  the  survey  respondents  to  their  process  performance  modeling  efforts.  The  same  is  so  for 
the  use  of  a  variety  of  modeling  and  optimization  techniques.  However,  as  shown  in  Figure  24,  the 
lead-in  question  asked  simply  whether  or  not  the  methods  were  used  at  all,  and  the  use  of  the  tech¬ 
niques  was  much  less  common  than  are  the  statistical  techniques  sununarized  in  Figure  23.  In  fact,  over 
20  percent  of  the  respondents  use  none  of  these  techniques. 


To  what  extent  are  the  following  statistical  methods  used 
in  your  organization’s  process  performance  modeling? 
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□  Don't  know 

■  Little  If  any 
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Figure  23:  Use  of  diverse  statistical  methods 


For  comparison  with  an  earlier  survey  of  high  maturity  organizations,  see  Crosstalk  [Goldenson  2003a]. 
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Which  of  the  following  other  optimization  approaches  are  used 
in  your  organization’s  process  performance  modeling? 
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40% 


20% 
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Figure  24:  Use  of  optimization  techniques 


The  questionnaire  also  queried  the  respondents  about  the  use  in  their  organizations  of  a  variety  of  deci¬ 
sion  and  display  techniques.  These  questions  were  included  largely  to  track  the  adoption  profile  by  the 
community  across  time  and  to  introduce  the  survey  respondents  who  were  not  yet  familiar  with  them  to 
a  full  range  of  techniques  that  might  be  used  profitably  in  their  own  organizations.  As  shown  in  Figure 
25,  relatively  large  numbers  of  the  responding  organizations  report  using  decision  trees,  weighted  multi 
criteria  methods,  and  wide  band  Delphi;  however  the  other  techniques  are  not  widely  used.  In  contrast, 
the  results  in  Figure  26  show  that  much  larger  numbers  of  these  organizations  use  a  series  of  common 
visual  display  techniques.  It  is  somewhat  surprising  that  over  half  report  using  box  plots.  Note  however 
that  the  exception  is  categorical  mosaic  charts,  such  as  those  used  in  Section  6  of  this  report.  Such  vis¬ 
ual  displays  are  appropriate  for  categorical  and  ordered  data,  which  also  are  quite  common  in  manage¬ 
ment  environments  and  software  and  systems  engineering. 
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Which  of  these  decision  techniques  are  used  in  your  organization? 


Figure  25:  Use  of  decision  techniques 


Which  of  the  foiiowing  visuai  dispiav  techniques  are  used 
to  communicate  the  resuits  of  your  organization’s 
anaiyses  of  process  performance  baseiines? 


Figure  26:  Use  of  visuai  display  techniques 

Two  other  sets  of  analytic  questions  were  included  in  the  survey  questionnaire.  The  rationale  for  both 
consistent  with  observations  by  SET  technical  staff  and  others  as  well  as  best  practice  in  measurement 
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and  analysis.  Organizations  in  our  experience  have  reported  difficulties  with  the  manual  collection  of 
data,  often  including  significant  challenges  to  maintaining  the  quality  and  integrity  of  the  data  that  are 
collected.  The  automation  questions  refer  to  a  growing  list  of  different  types  of  automation,  ranging 
from  workflow  automation  to  statistical  analysis  packages.  The  results  are  summarized  in  Figure  27. 
Reliance  on  spreadsheets,  automated  data  collection,  and  management  software  was  relatively  exten¬ 
sive;  however,  the  same  is  not  yet  so  for  statistical  packages,  workflow  automation,  or  report  prepara¬ 
tion  software. 


The  reported  use  of  methods  to  ensure  data  quality  and  integrity  that  are  summarized  in  Figure  28  sug¬ 
gest  that  these  high  maturity  organizations  pay  reasonably  close  attention  to  these  important  matters. 
Still,  the  proportions  of  frequent  use  drop  off  considerably  to  the  right  side  of  the  Pareto  distribution. 
There  is  room  for  continuous  improvement  here  as  well. 


Moreover,  as  shown  in  Section  6,  both  of  these  question  sets  are  moderately  strongly  related  statisti¬ 
cally  with  the  respondents’  reports  of  the  value  added  by  their  organizations’  process  performance 
modeling.  The  results  will  be  used  to  help  guide  future  SEI  training  and  coaching.  Other  current  work 
at  the  SEI  focuses  on  these  issues  as  well  [Kasunic  2008],  and  we  may  focus  on  the  topic  in  more  detail 
in  future  state  of  the  practice  surveys. 


How  much  automated  support  is  available  for  measurement 
related  activities  in  your  organization? 


□  Don't  know 

□  Does  not  apply 

■  Little  if  any 

□  Limited 

□  Moderate 

■  Substantial 

□  Extensive 


Figure  27:  Use  of  automated  support  for  measurement  and  analysis  activities 
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How  often  does  your  organization  do  the  foiiowing 
[quaiityand  data  integrity  checks]  with  the  data  it  coiiects? 


Figure  28:  Use  of  methods  to  ensure  data  quality  and  integrity 


□  Don't  know 

■  Rarely  if  ever 

□  Occasionally 

□  About  half  the  time 

■  Frequently 

□  Almost  always 
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5  Baselining  High  Maturity  Organizational  Context 


The  survey  respondents’  reports  about  the  extent  to  which  their  organizational  contexts  differed  are 
described  here.  These  include  management  support,  staffing,  and  resources;  differences  in  the  technical 
challenges  faced  by  their  projects;  and  other  barriers  and  facilitators  to  successful  use  of  process  per¬ 
formance  modeling. 

5.1  Management  Support,  Staffing,  and  Resources 

The  importance  of  management  support  for  process  improvement  is  widely  acknowledged.  There  is 
empirical  evidence  that  the  existence  of  such  support  can  increase  the  likelihood  of  the  snccess  of 
measurement  and  analysis  programs  in  general  [Goldenson  1999,  Goldenson  2000,  Gopal  2002]. 

In  order  for  management  to  provide  truly  informed  support  for  the  creation  and  use  of  process  perform¬ 
ance  baselines  and  models,  they  must  understand  the  results  presented.  The  answers  to  such  a  question 
are  summarized  in  Figure  29.  As  shown  in  the  figure,  the  modal  category  (46  percent)  is  “moderately 
well.”  That  and  the  fact  that  over  10  percent  more  of  the  respondents  reported  less  understanding  by 
management  users  of  their  model  results  provides  confidence  on  face  validity  grounds  that  these  re¬ 
spondents  are  replying  candidly. 

Flowever,  note  that  over  40  percent  of  the  respondents  reported  that  managers  in  their  organizations 
understood  the  model  results  very  well  or  even  extremely  well.  As  described  in  Section  6,  answers  to 
this  question  varied  consistently  with  the  respondents’  independently  recorded  answers  about  the  ef¬ 
fects  of  using  their  organizations’  process  performance  models.  Answers  to  this  and  similar  questions 
will  be  profiled  across  time  to  evaluate  the  growth  in  confidence  that  management  nsers  have  in  under¬ 
standing  the  results  of  process  performance  baselines  and  models. 

The  availability  of  qualified  and  well-prepared  individuals  to  do  process  performance  modeling  is  an¬ 
other  important  measure  of  management  support.  As  shown  in  Figure  30,  most  of  the  respondents  said 
that  such  individuals  are  available  when  needed  on  at  least  a  frequent  basis;  however,  over  thirty  per¬ 
cent  experience  difficulty  in  obtaining  such  expertise. 
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How  well  do  the  managers  In  your  organization  who  use  process 
performance  model  results  understand  the  results  that  they  use? 


Figure  29:  Level  of  understanding  of  process  performance  model  results  attributed  to  managers 


How  often  are  qualified,  well-prepared  people  available  to  work  on 
process  performance  modeling  in  your  organization  when  you  need  them? 


Frequently 

32% 


N  =  149 


Figure  30:  Availability  of  qualified  process  performance  modeling  personnel 

We  also  asked  a  question  in  this  context  about  how  measurement  and  analysis  is  staffed  in  the  survey 
respondents’  organizations.  The  results  are  summarized  in  Figure  31.  The  bulk  of  these  respondents 
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said  that  their  organizations  relied  on  organization-wide  groups  or  corporate  snpport  groups  such  as 
engineering  process,  quality  assurance,  or  measurement  groups.  However,  there  are  discrepancies  be¬ 
tween  these  results  and  those  from  other  SEI  surveys.  Results  from  the  SEFs  2007  general  population 
state  of  the  measurement  and  analysis  practice  survey  found  that  higher  maturity  organizations  were 
more  likely  to  rely  on  staff  from  projects  and  similar  organizational  units  throughout  their  organizations 
[Goldenson  2007b].  Such  a  result  is  consistent  with  the  existence  of  widespread  measurement  expertise 
throughout  such  organizations. 

It  is  possible  that  the  respondents  in  the  current  survey  answered  the  qnestion  from  a  different  perspec¬ 
tive  since  they  knew  that  they  were  chosen  because  of  their  high  maturity  status.  However,  unpublished 
results  from  the  2008  SEI  general  population  survey  are  similar  to  the  high  maturity  results  shown  here. 
Sampling  bias  may  be  a  factor,  but  the  question  wording  itself  also  needs  to  be  made  clearer  in  future 
series.  Since  there  is  some  question  about  the  generalizability  of  these  results,  relationships  with  the 
respondents’  reports  about  the  overall  value  of  their  process  performance  modeling  efforts  are  not 
shown  in  this  report. 


Which  of  the  following  best  describes  how  work  on  measurement  and  analysis  is 
staffed  in  your  organization? 


A  few  key 
measurement 
experts 


Other 


Groups  or  individuals 
in  different  projects 
or  organizational 
units 
15% 


Organization-wide 

76% 


N  =  144 


Figure  31:  Staffing  of  measurement  and  anaiysis  in  responding  organizations 


We  also  asked  the  survey  respondents  a  short  series  of  questions  about  how  well  the  builders  and  main- 
tainers  of  their  process  performance  models  understand  the  CMMI  definitions  and  concepts  surronnd- 
ing  process  performance  baselines  and  models.  Notice  in  Eigure  32  that  over  80  percent  of  the  respon¬ 
dents  said  that  their  model  builders  and  maintainers  understood  the  CMMI  definitions  of  process 
performance  baselines  and  models  very  well  or  better.  However,  their  perceptions  of  the  modelers’  un¬ 
derstanding  of  the  circumstances  under  which  such  baselines  and  models  were  likely  to  be  usefnl  are 
somewhat  lower. 
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Knowing  that  a  number  of  misconceptions  about  baselines  and  models  exist  in  the  community,  these 
questions  serve  to  evaluate  the  current  self-perception  of  a  knowledge  or  training  gap.  Admittedly,  this 
question  series  convolutes  the  recognition  of  knowledge  with  the  unrecognized  lack  of  knowledge. 
Hopefully  profiling  the  answers  to  this  question  across  time  will  reflect  a  growing  confidence  by  the 
community  in  its  understanding  of  process  performance  baselines  and  models. 


How  well  do  the  people  who  create  your  organization’s  process  performance 
models  and  baselines  understand  the  intent  of  CMMI? 
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□  Moderately  well 

□  Very  well 

□  Extremely  well 


The  CMMI 
definition  of  a 
process 
performance 
model 


The  circumstances 
when  process 
performance 
baselines  are 
useful 


The  CMMI 
definition  of  a 
process 
performance 
baseline 


The  circumstances 
when  process 
performance 
models  are  useful 


N  =  148 


Figure  32:  Level  of  understanding  of  CMMI  Intent  attributed  to  modelers 

To  what  extent  does  the  mixed  use  of  incentives  matter  for  the  outcomes  of  quality,  performance,  or 
process  improvements?  While  differences  in  accepted  labor  practices  by  domain  and  industry  will  af¬ 
fect  the  results,  can  significant  incentives  and  rewards  play  a  vital  role  in  the  adoption  and  use  of  meas¬ 
urement  and  modeling  that  dramatically  improve  business  results?  To  address  such  issues,  we  asked  a 
series  of  questions  meant  to  provide  insight  into  the  promotional  or  financial  incentives  that  are  tied  to 
the  deployment  and  adoption  of  measurement  and  analysis  in  the  respondents’  organizations.  The  re¬ 
sults  are  summarized  in  Figure  33.  As  shown  in  the  figure,  almost  half  of  the  survey  respondents  re¬ 
ported  that  no  such  incentives  exist  in  their  organizations;  however,  the  others  did  report  the  use  of 
such  incentives  for  one  or  more  of  their  employee  groups.  Given  the  loose  coupling  of  this  question 
series  to  process  performance  modeling  per  se,  we  have  not  addressed  this  topic  further  in  this  report. 
However,  we  do  intend  to  track  such  questions  over  time  and  in  more  detail  elsewhere. 
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Does  your  organization  provide  promotion  or 
financiai  incentives  for  its  employees  that  are  tied  to  the 
deployment  and  adoption  of  measurement  and  analysis? 


For  project 

For  project 

For  middle 

For  For  others 

managers 

engineers 

managers 

executive 

and  other 

(e.g., 

and  senior 

technical 

program  or 

managers 

staff 

product  line) 

Figure  33:  Promotional  incentives  for  measurement  and  analysis  in  responding  organizations 

5.2  Technical  Challenge 

This  set  of  questions  asked  the  respondents  about  a  series  of  technical  challenges  that  may  have  been 
faced  by  the  projects  and  product  teams  in  their  organizations.  The  individual  questions  focused  most 
heavily  on  product  characteristics,  although  some  relate  more  directly  to  organizational  context.  The 
concern  is  that  different  degrees  of  technical  challenge  can  warrant  different  degrees  of  sophistication 
in  measurement,  analytic  methods,  and  predictive  modeling.  Previous  work  has  suggested  that  such 
challenges  can  directly  affect  the  chances  of  project  success  independently  of  project  capability  [Elm 
2008,  Ferguson  2007]. 

As  shown  in  Figure  34,  issues  with  respect  to  extensive  interoperability,  large  amounts  of  development 
effort,  and  quality  constraints  presented  the  greatest  difficulty  from  the  respondents’  perspectives. 
However,  as  will  be  shown  in  Section  6,  the  overall  level  of  technical  challenge  reported  by  these  re¬ 
spondents  remained  relatively  low. 
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Following  are  a  series  of  statements  about  the  kinds  of  technical  chalienqes 
that  projects  sometimes  face.  How  weii  do  they  describe  vour  organization? 


1 00% 

80% 

60% 

40% 

20% 

0% 


□  Not  applicable 

□  Don't  know 
■  Hardly  at  all 

□  To  a  limited  extent 

□  To  a  moderate  extent 
-|  BTo  a  large  extent 

□  Almost  entirely 


Figure  34:  Project  technical  challenge  in  responding  organizations 


5.3  Barriers  and  Facilitators 

The  survey  questionnaire  included  a  battery  of  questions  about  the  extent  to  which  responding  organi¬ 
zations  experienced  several  common  situations  in  doing  their  process  performance  modeling.  Some  of 
the  questions  were  stated  negatively  as  management  and  analytic  barriers  or  obstacles  to  doing  such 
work.  Others  were  stated  positively  as  management  and  analytic  facilitators  or  enablers  of  successful 
baselining  and  modeling  efforts.  The  results  may  serve  to  forewarn  organizations  as  well  as  help  guide 
and  prioritize  the  process  performance  coaching  provided  by  the  SEI  and  others.  Composite  variables 
based  on  these  same  questions  were  used  in  Section  6  to  help  explain  the  variation  in  the  degree  of 
business  benefit  attributed  to  the  use  of  the  participating  organizations’  process  performance  models.^ 

Notice  in  Figure  35  that  almost  50  percent  of  the  survey  respondents  reported  that  their  organizations 
have  experienced  difficulty  to  at  least  a  moderate  extent  in  doing  process  performance  modeling  be¬ 
cause  of  the  time  that  it  takes  to  accumulate  enough  historical  data.  A  comparable  number  of  the  re¬ 
spondents  reported  that  their  organizations  experienced  difficulty  because  their  managers  were  less 
willing  to  fund  new  work  when  the  outcome  was  uncertain. 


In  addition  to  the  distinction  shown  here  between  barriers  and  facilitators,  the  individual  items  also  were  grouped 
separately  as  aspects  of  management  support  and  use  of  exemplary  modeling  approaches  for  the  analyses  in  Section 
6. 

While  this  can  present  considerable  difficulty,  modeling  and  simulation  can  provide  useful  solutions  in  the  absence  of 
such  information. 
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Following  is  a  series  of  statements  that  are  made  in  some 
organizations  about  the  use  of  process  performance  modeling. 
How  well  do  they  describe  vour  organization?  [Stated  as  problems] 


□  Not  applicable 

□  Don't  know 

■  Hardly  at  all 

□  To  a  limited  extent 

□  To  a  moderate  extent 

■  To  a  large  extent 

□  Almost  entirely 
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Figure  35:  Management  &  analytic  barriers  to  effective  measurement  and  analysis 

Figure  36  shows  comparable  results  for  facilitators  of  successful  process  performance  modeling.  On  the 
positive  side,  over  70  percent  of  the  respondents  reported  that  their  organizations  have  been  made  bet¬ 
ter  because  their  managers  wanted  to  know  when  things  were  off  track.  However,  none  of  the  other 
situations  were  widely  identified  by  the  respondents  as  management  and  analytic  facilitators  or  enablers 
of  successful  baselining  and  modeling  efforts  in  their  organizations. 


A  second  battery  of  questions  asked  about  major  obstacles  that  may  have  inhibited  progress  during  the 
respondents’  organizations' journeys  to  high  maturity.  As  shown  in  Figure  37,  several  of  these  potential 
difficulties  were  experienced  by  20  percent  or  more  of  the  organizations,  and  barely  20  percent  re¬ 
ported  not  having  experienced  any  of  them.  Many  of  the  questions  in  this  series  overlap  with  others  in 
the  survey,  and  others  are  only  loosely  coupled  to  process  performance  modeling  per  se.  Hence  we 
have  not  included  them  in  the  analysis  in  Section  6.  However  they  may  prove  to  be  worth  clarifying 
and  tracking  over  time. 
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Following  Is  a  series  of  statements  that  are  made  in  some 
organizations  about  the  use  of  process  performance  modeling. 
How  well  do  they  describe  vour  organization?  [Stated  as  strengths] 


□  Not  applicable 

□  Don't  know 

■  Hardly  at  all 

□  To  a  limited  extent 

□  To  a  moderate  extent 

■  To  a  large  extent 
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Figure  36:  Management  &  analytic  facilitators  of  effective  measurement  and  analysis 


Which,  if  any,  of  the  following  have  been  major  obstacles 
during  your  organization’s  journey  to  high  maturity? 


Figure  37:  Major  obstacles  to  achieving  high  maturity 
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6  Explaining  Variability  in  Organizational  Effects  of  Process 
Performance  Modeling 


Results  describing  the  degree  to  which  the  respondents’  reports  about  the  value  of  process  performance 
modeling  to  their  organizations  varied  as  a  function  of  their  answers  to  the  survey  questions  about  the 
various  process  and  organizational  contextual  factors  discussed  earlier  in  this  report.  Those  results  are 
described  in  this  section.  As  noted  in  Section  3,  we  use  the  question  about  the  overall  effect  attributed 
by  the  respondents  to  their  organizations’  modeling  efforts  as  the  “big  Y”  factor  here.  The  questions 
about  specific  effects  of  the  modeling  are  not  all  necessarily  equally  pertinent  to  all  of  the  surveyed 
organizations;  however,  similar  results  to  those  based  on  the  overall  question  also  exist  using  a  com¬ 
posite  measure  of  the  specific  effects.  Some  of  the  other  questions  that  are  used  as  x  variables  also  are 
used  or  discussed  here  as  interim  y  variables. 

6.1  Analysis  Methods  Used  in  this  Report 
Summarizing  the  results 

Most  of  the  results  described  in  this  section  summarize  relationships  between  two  variables."  They  are 
described  using  a  graphical  mosaic  representation  that  shows  in  an  intuitive  visual  manner  the  extent  to 
which  the  survey  respondents’  answers  vary  in  a  consistent  manner.  An  example  can  be  seen  in  Figure  38. 
The  data  for  this  example  come  from  the  2007  SEI  state  of  measurement  and  analysis  survey  [Goldenson 
2008b]. 

Notice  that  the  maturity  level  values  are  displayed  along  the  horizontal  x  axis  of  the  mosaic  and  the  vari¬ 
able  name  is  displayed  below  it.  Labels  for  the  respondents’  answer  to  a  question  about  how  frequently 
measurement  and  analysis  activities  took  place  in  their  organizations  are  displayed  to  the  right  of  the  mo¬ 
saic  on  the  vertical  y  axis;  the  variable  name  used  in  the  statistical  software  is  shown  on  the  left  side  of  the 
mosaic.  The  proportions  of  responses  about  frequency  of  measurement  use  for  the  organizations  at  each 
maturity  level  are  shown  in  separate  columns  of  the  mosaic,  where  each  value  of  the  y- variable  is  repre¬ 
sented  in  a  separate  mosaic  tile.  The  percentages  represented  by  -  0  to  1  also  appear  to  the  left  of  the  full 
mosaic  and  correspond  to  the  heights  of  the  tiles.  A  separate,  single  column  mosaic  to  the  right  shows  the 
total  of  all  the  responses  about  frequency  of  use,  regardless  of  maturity  level.  It  also  serves  as  a  legend 
that  visually  identifies  each  value  of  the  ordinal  y  variable  in  the  corresponding  x  variable  columns.  Note 
that  the  value  labels  for  the  y- variable  are  aligned  with  the  total  distribution  in  the  single  column  mosaic 
that  serves  as  a  legend  for  the  comparison  with  the  x- variable  in  the  full  mosaic. 

As  shown  in  the  figure,  the  organizations  at  maturity  level  one  were  much  less  likely  to  report  using 
measurement  and  analysis  on  a  routine  basis,  although  over  a  quarter  of  them  do  claimed  to  do  so.  Not 
surprisingly,  the  proportions  of  organizations  that  reported  being  routine  measurement  users  increase  in  a 
stair-step  pattern  along  with  increasing  maturity  level,  with  the  biggest  difference  between  level  one  and 
two.  Notice  also  that  the  width  of  each  column  varies  in  proportion  to  the  number  of  responding  organiza¬ 
tions  at  each  maturity  level.  This  can  provide  a  quick  sense  of  how  evenly  or  unevenly  the  survey  re¬ 
sponses  are  distributed.  In  this  instance  there  are  quite  a  few  more  organizations  at  maturity  level  one. 
(Maturity  levels  four  and  five  are  combined  since  almost  all  of  these  organizations  say  they  are  at  level  5.) 

”  Brief  mention  of  iogistic  regression  for  muitipie  x-variabie  comparisons  can  be  found  in  Section  6.7. 
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Maturity  level 


Gamma  =  .73;  p  <  .00001 ; 


N  =  365 


Figure  38:  Example:  relationship  between  maturity  level  and  frequency  of  use  of 
measurement  and  analysis,  from  2007  state  of  the  practice  survey 


The  overall  strength  of  the  relationship  between  the  two  variables  can  be  described  by  the  value  of  the 
gamma  statistic.  Goodman  and  Kruskal’s  gamma  is  an  ordinal  measure  of  association  that  is  appropri- 
ate  for  ordered  categorical  measures  such  as  those  that  are  used  in  this  report.  It  is  symmetric,  which 
means  that  its  value  will  be  the  same  regardless  of  which  variable  is  considered  to  be  an  x  factor  or  a  y 
factor.  The  value  of  gamma  represents  the  difference  between  concordant  and  discordant  pairs  of  val¬ 
ues  on  two  variables.  It  is  computed  as  the  excess  of  concordant  pairs  as  a  percentage  of  all  pairs,  ig¬ 
noring  ties.  The  notion  of  concordance  for  any  pair  of  values  means  that  as  the  x  value  increases  its 
corresponding  y  value  also  must  increase  (or  decrease  for  negative  relationships).  Gamma  is  based  on 
weak  monotonicity  (i.e.,  ignoring  ties  means  that  the  y  value  can  remain  the  same  rather  than  increase) 


Similar  to  many  other  correlation  coefficients  and  measures  of  association,  gamma  varies  from  -1  to  1. 
Values  of  0  indicate  statistical  independence  (no  relationship)  and  values  of  1  indicate  perfect  relation¬ 
ships  (-1  is  a  perfect  negative  relationship,  where  values  on  one  variable  decrease  while  the  other  in¬ 
creases).  Gamma  is  a  proportional  reduction  in  error  (PRE)  statistic  with  an  intuitive  interpretation. 
Conceptually  similar  to  Pearson’s  r  for  interval  or  ratio  data,  the  value  of  gamma  is  simply  the  propor¬ 
tion  of  paired  comparisons  where  knowing  the  rank  order  of  one  variable  allows  one  to  predict  accu¬ 
rately  the  rank  order  of  the  other  variable. 

The  p  value  in  the  figure  is  the  result  of  a  statistical  test  of  the  likelihood  that  a  concordant  relationship 
of  gamma’ s  magnitude  could  have  occurred  by  chance  alone.  In  this  instance  the  chances  that  a  gamma 
of  .73  would  occur  simply  by  random  chance  is  less  than  one  in  100,000.  The  relationship  is  based  on 
the  number  (N)  of  survey  respondents  (365)  who  answered  both  questions.  By  convention,  p  values 
less  than  or  equal  to  .05  are  considered  statistically  significant. 


A  clear  description  may  be  found  in  Linton  Freeman’s  now  classic  basic  text  [Freeman  1965]. 
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Composite  measures 


Many  of  the  relationships  described  in  this  report  use  composite  measures  that  are  based  on  combina¬ 
tions  of  several  related  component  questions.  The  measure  of  emphasis  on  “healthy  process  perform¬ 
ance  model  ingredients”  shown  in  Figure  39  on  page  38  is  one  such  composite.  It  is  based  on  a  combi¬ 
nation  of  the  respondents’  answers  to  the  group  of  related  questions  described  in  Figure  15  on  page  15 
that  asked  about  the  extent  to  which  their  organizations’  process  performance  models  were  used  for 
various  purposes.  The  possible  answers  to  those  questions  include  “extensive,”  “substantial,”  “moder¬ 
ate,”  “limited”  and  “little  if  any.”‘^ 

Like  most  of  the  other  composite  measures  used  in  this  report,  this  one  is  a  weighted,  summed  index  of 
the  respondents’  answers  to  each  of  those  questions. Much  like  a  grade-point  average,  the  answers  are 
assigned,  ordered  numeric  values  that  are  simply  added  and  then  divided  by  the  number  of  valid  an¬ 
swers  to  the  series  of  questions  for  each  respondent.*^  For  example  in  Figure  39,  “extensive”  answers 
are  scored  as  the  value  5,  “substantial”  as  4,  down  to  “little  if  any”  as  1.  The  index  scores  are  then  sepa¬ 
rated  into  the  categories  shown  on  the  figures’  x  or  y  axes  based  on  the  distribution  of  response  values. 

The  category  cutting  points  are  set  based  on  the  closeness  to  the  component  questions’  response  catego¬ 
ries  and  ensuring  that  there  are  enough  cases  in  each  category  for  meaningful  analysis.  In  Figure  39,  the 
lowest  category  (“<  Moderate”)  includes  the  composite  scores  with  values  less  than  3.  The  second 
category  (“Moderate  to  <  midway  toward  substantial”)  includes  values  that  range  from  3  to  less  than 
3.5.  The  third  category  (“Toward  but  <  substantial”)  ranges  from  3.5  to  less  than  4.  The  highest  cate¬ 
gory  (“Substantial  or  better”)  includes  composite  scores  that  are  equal  to  or  greater  than  4. 

There  are  several  reasons  to  combine  the  component  questions  into  single  composite  indices.  Of 
course,  reducing  the  number  simplifies  the  analysis.  While  it  may  seem  counterintuitive,  combining  the 
components  also  follows  a  basic  reliability  principle.  There  always  is  noise  in  survey  data  (actually  in 
measured  data  of  any  kind).  Respondents  can  be  uncertain  about  their  answers  concerning  the  details  of 
a  specific  question,  or  the  lack  of  clarity  in  the  wording  of  a  specific  question  may  cause  different  re¬ 
spondents  to  attribute  different  meanings  to  the  same  question.  Other  things  being  equal,  the  unreliabil¬ 
ity  can  be  averaged  out  such  that  the  composite  index  is  more  reliable  than  many  or  all  of  its  individual 
components  [Hill  2006,  Coleman  1964,  Guilford  1954]. 

Interpretation 

Survey  data  such  as  these  do  not  speak  for  themselves.  Interpretation  is  necessary  for  all  statistical 
analyses,  including  those  based  on  controlled  experiments.  Perceptions  and  expectations  often  differ 
among  survey  respondents  and  maturity  levels.  Moreover,  survey  data  such  as  these  often  are  collected 
at  a  single  point  in  time.  It  is  difficult  to  separate  cause  and  effect,  which  often  are  reciprocal  over  time. 
Proportions  and  strength  of  association  sometimes  vary  in  subtle  ways.  Still,  the  differences  described 


The  very  few  answers  of  “don’t  know”  and  “does  not  apply”  were  excluded  from  the  calculations.  Answers  to  the 
“other”  categories  that  sometimes  exist  In  related  question  sets  also  were  Included  In  the  composite  Indices. 

In  other  Instances,  the  composite  variables  were  simply  counts  of  the  numbers  of  check  boxes  the  respondents  se¬ 
lected. 

The  weighting  and  summing  are  mathematically  equivalent  to  an  arithmetic  mean;  however,  also  much  like  a  grade 
point  average,  the  results  simply  are  rank  orders.  Such  Indices  are  not  Interval-  or  ratio-level  measures  that  can  be 
added  or  multiplied  meaningfully. 
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in  this  report  are  consistent  with  what  we  think  we  know  about  process  maturity  and  measnrement 
practice. 

6.2  Process  Performance  Modeling  and  Analytic  Methods 

As  shown  in  Figure  39,  there  is  a  very  strong  empirical  relationship  between  the  value  the  respondents 
attribute  to  their  organizations’  process  performance  modeling  efforts  and  the  degree  to  which  that 
modeling  is  consistent  with  the  emphases  on  the  healthy  ingredients  of  such  models.  Those  whose  or¬ 
ganizations  placed  greater  emphasis  on  controllable  and  uncontrollable  factors,  detailed  subprocesses 
and  more  broadly  defined  processes,  modeling  uncertainty  and  variability — along  with  important  seg¬ 
menting  factors — were  considerably  more  likely  to  attribute  more  value  to  their  modeling  efforts.  Note 
also  by  the  width  of  the  columns  that  there  was  considerable  room  for  improving  such  emphasis,  which 
may  add  considerable  business  value. 

A  similar  result  is  shown  in  Figure  40,  which  is  based  on  the  respondents’  reports  about  the  uses  to 
which  process  performance  models  were  put  in  their  organizations.  This  relationship  is  somewhat 
stronger  than  the  first  one.  Equally  noteworthy,  more  organizations  appear  to  have  implemented  their 
models  consistently  with  the  healthy  ingredients,  as  seen  by  the  column  widths  in  the  figure.  They  were 
considerably  more  likely  to  use  their  models  to  predict  both  interim  and  final  process  ontcomes,  iden¬ 
tify  the  need  for  mid-course  corrections,  model  expected  variability  of  outcomes,  and  enable  what-if 
analyses. 


Figure  39:  Relationship  between  emphasis  on  healthy  process  performance  model 
ingredients  and  overall  value  attributed  to  process  performance  models 
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Figure  40:  Relationship  between  use  of  healthy  process  performance  model 

ingredients  and  overall  value  attributed  to  process  performance  models 


A  similarly  strong  relationship  is  shown  in  Figure  41.  It  shows  the  relationship  between  overall  value 
attributed  to  process  performance  modeling  and  the  diversity  of  models  used  to  predict  product  quality 
and  project  performance.  These  include  delivered  defects,  cost  and  schedule  duration,  accuracy  of  es¬ 
timates,  type  and  severity  of  defects,  quality  of  services  provided,  customer  satisfaction,  product  quality 
attributes,  work  product  size,  and  measures  of  ROI  or  financial  performance.  As  shown  in  the  figure, 
organizations  that  maintained  and  used  a  richer  and  more  varied  suite  of  process  performance  models  to 
predict  product  quality  and  project  performance  were  much  more  likely  to  find  value  in  their  modeling 
than  those  that  did  not.'® 


A  similar  result  exists  for  the  diversity  of  models  of  interim  performance  outcomes  (e.g.,  estimates  at  completion,  es¬ 
caped  defects,  and  the  other  classes  of  measures  described  in  Figure  20  on  page  1 9);  however,  the  relationship  with 
diversity  of  models  of  interim  performance  outcomes  as  the  x  factor  is  only  moderately  strong  (gamma  =  .36).  These 
kinds  of  models  appear  to  have  less  consistent  value  for  the  surveyed  organizations. 
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Figure  41:  Relationship  between  diversity  of  models  used  (for  predicting  product  quality 

and  project  performance)  and  overall  value  attributed  to  process  performance  models 

As  mentioned  in  Section  5.3,  we  grouped  some  of  the  questions  about  barriers  and  facilitators  of  proc¬ 
ess  performance  modeling  into  a  composite  measure  of  the  use  of  exemplary  modeling  approaches.  The 
following  items  were  used  to  construct  that  measure: 

•  'We  have  trouble  doing  process  performance  modeling  because  it  takes  too  long  to  accumulate 
enough  historical  data. 

•  We  thought  we  knew  what  was  driving  process  performance,  but  process  performance  modeling 
has  taught  us  otherwise. 

•  We  use  data  mining  when  similar  but  not  identical  electronic  records  exist. 

•  We  do  real  time  sampling  of  current  processes  when  historical  data  are  not  available. 

•  We  create  our  baselines  from  paper  records  for  previously  unmeasured  attributes. 

The  relationship  shown  in  Figure  42  is  a  strong  one.  Note  that  relatively  few  respondents  said  that  such 
approaches  described  the  situations  in  their  organizations  to  even  a  moderate  extent.  We  expect  the 
relationship  to  become  stronger  as  more  organizations  adopt  such  approaches  over  time  and  gain  value 
from  doing  so. 


This  item  was  reverse  scored  for  inciusion  into  the  composite  index  since  it  is  stated  negativeiy  as  a  barrier  rather  than 
a  faciiitator  of  process  performance  modeiing. 
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Gamma  =  .48;  p  <  .00002; 
N  =  143 


Figure  42:  Relationship  between  use  of  exemplary  modeling  approaches  and  overall 
value  attributed  to  process  performance  models 


A  richer  set  of  analytic  methods  also  appears  to  pay  off  for  these  organizations.  Another  quite  strong 
relationship  exists  in  Figure  43  between  overall  value  attributed  to  process  performance  modeling  and  a 
composite  measure  based  on  the  mix  of  different  statistical  methods  used  by  these  organizations.  Recall 
that  these  include  the  following: 

•  regression  analysis  predicting  continuous  outcomes  (e.g.,  bivariate  or  multivariate  linear  regres¬ 
sion  or  non-linear  regression) 

•  regression  analysis  predicting  categorical  outcomes  (e.g.,  logistic  regression  or  loglinear  models) 

•  analysis  of  variance  (e.g.,  ANOVA,  ANCOVA,  or  MANOVA) 

•  attribute  SPC  charts  (e.g.,  c,  u,  p,  or  np) 

•  individual  point  SPC  charts  (e.g.,  ImR  or  XmR) 

•  continuous  SPC  charts  (e.g.,  XbarR  or  XbarS) 

•  design  of  experiments 

Once  again,  the  width  of  the  leftmost  column  shows  room  for  increasing  the  use  of  appropriate  statisti¬ 
cal  methods.  The  greater  likelihood  of  finding  value  as  one  moves  to  the  right  in  the  mosaic  also  sug- 

1 8 

gests  that  improvement  in  an  organization’s  statistical  capabilities  can  be  well  worth  the  effort. 


Regression  and  ANOVA  are  the  best  Individual  discriminators. 
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Figure  43:  Relationship  between  use  of  statistical  methods  and 
overall  value  attributed  to  process  performance  models 


There  also  is  a  moderately  strong  to  strong  relationship  between  use  of  optimization  methods  and  value 
attributed  to  the  organizations’  process  performance  modeling.  Recall  that  these  include  the  following: 

•  Monte  Carlo  simulation 

•  discrete  event  simulation  for  process  modeling 

•  Markov  or  Petri-net  models 

•  probabilistic  modeling 

•  neural  networks 

•  optimization 

As  shown  in  Figure  44,  the  strength  of  the  relationship  may  be  attenuated  because  so  few  of  the  re¬ 
sponding  organizations  used  more  than  one  of  these  methods.  However,  the  result  does  suggest  that 
more  use  of  such  analytic  methods  will  prove  to  be  worth  the  effort. 

As  shown  in  Figure  45  and  Figure  46,  comparably  strong  relationships  also  exist  for  the  use  of  auto¬ 
mated  support  for  measurement  and  analysis  and  the  use  of  data  quality  and  integrity  checks.  As  noted 
in  Section  4.4  and  can  be  seen  in  the  width  of  the  bars  in  Figure  45,  there  is  considerable  room  for  in¬ 
creasing  automation  in  these  organizations,  but  it  is  likely  to  pay  off  in  better  modeling  outcomes.  The 
X  factor  categories  are  much  more  evenly  distributed  in  Figure  46,  and  all  except  those  in  the  leftmost 
category  are  relatively  likely  to  check  their  data  for  quality  and  integrity.  Yet  the  overall  pattern  in  the 
mosaic  is  quite  consistent,  with  clear  stair-step  patterns  of  increases  from  left  to  right  in  the  proportions 
who  find  their  modeling  efforts  to  be  extremely  valuable  and  decreases  in  those  who  report  mixed 
value  or  worse. 
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Use  of  optimization  methods  (S5Q3) 


Gamma  =  .44;  p  <  .0004; 
N  =  143 


Figure  44:  Relationship  between  use  of  optimization  methods  and  overall  value 
attributed  to  process  performance  models 
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Extremely  valuable 


Very  valuable 
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Gamma  =  .42;  p  <  .0002; 
N  =  144 


Figure  45:  Relationship  between  automated  support  for  measurement  and  analysis  and 
overall  value  attributed  to  process  performance  models 
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Figure  46:  Relationship  between  data  quality  and  integrity  checks  and  overall  value 
attributed  to  process  performance  models 

Finally,  notice  the  very  strong  relationship  in  Figure  47.  The  overall  value  attrihuted  to  process  per¬ 
formance  modeling  varies  quite  predictably  with  the  extent  to  which  model  results  are  used  to  inform 
decision  making  in  status  and  milestone  reviews.  The  extent  to  which  the  model  results  were  used  un¬ 
doubtedly  is  a  function  of  management  support  and  the  pertinence  of  the  results  as  well  as  the  quality 
of  the  models  themselves  and  the  analytic  methods  used  to  build  them.  Not  surprisingly,  this  question 
turns  out  to  be  more  closely  related  statistically  to  the  performance  outcomes  of  doing  process  perform¬ 
ance  modeling  than  any  other  x  variable  examined  in  this  report. 


One  can  argue  that  organizations  use  the  models  in  their  reviews  because  they  find  vaiue  in  them  rather  than  the 
other  way  around.  The  causal  direction  undoubtedly  is  reciprocal  over  time  to  some  extent  at  least.  The  same  is  true 
for  other  of  our  x  variables  as  well.  Yet,  as  can  be  seen  in  Section  6.6,  some  of  those  who  find  reiativeiy  more  overall 
value  from  their  modeling  activities  also  are  consistently  least  likely  to  use  them  in  their  reviews. 
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Gamma  =  .67;  p  <  .00001 ; 
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Figure  47:  Relationship  between  use  of  process  performance  model  predictions  in  status  and 
milestone  reviews  and  overall  value  attributed  to  process  performance  models 


6.3  Management  Support 

As  already  discussed  in  Section  5,  the  importance  of  management  support  for  process  improvement  is 
widely  acknowledged  in  our  field,  and  there  is  evidence  that  such  support  can  increase  the  likelihood  of 
the  success  of  organizational  measurement  efforts  [Goldenson  1999,  Goldenson  2000,  Gopal  2002]. 

We  examine  three  aspects  of  such  support  in  this  section. 

First  of  all,  we  asked  the  survey  respondents  how  well  the  managers  in  their  organizations  who  use 
process  performance  model  results  understood  the  results  that  they  use.  The  relationship  between  their 
answers  to  the  question  and  what  they  tell  us  elsewhere  in  the  questionnaire  about  how  useful  process 
performance  models  have  been  for  their  organizations  is  shown  in  Figure  48.  Their  answers  to  the  two 
questions  co-vary  quite  consistently,  and  the  relationship  is  a  very  strong  one. 
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Figure  48:  Relationship  between  managers’  understanding  of  model  results  and  overall  value 

attributed  to  process  performance  models 

Similar  to  the  measure  of  exemplary  modeling  approaches  shown  in  Figure  42,  we  also  grouped  some 
of  the  questions  about  barriers  and  facilitators  of  process  performance  modeling  into  a  composite 

measure  of  management  support  for  modeling.  The  following  items  were  nsed  to  construct  that  meas- 

20 

ure: 

•  Doing  process  performance  modeling  has  become  an  accepted  way  of  doing  business  here. 

•  We  make  our  decisions  about  the  models  we  build  without  sufficient  participation  by  management 
or  other  important  stakeholders. 

•  We  have  trouble  convincing  management  about  value  of  doing  process  performance  modeling. 

•  The  messenger  has  been  shot  for  delivering  bad  news  based  on  process  performance  model  pre¬ 
dictions. 

•  Our  managers  want  to  know  when  things  are  offtrack. 

•  Our  managers  are  less  willing  to  fund  new  work  when  the  outcome  is  uncertain. 

As  shown  in  Figure  49,  this  relationship  also  is  a  strong  one.  These  results  provide  further  evidence 
about  the  importance  of  management  support  for  successful  process  performance  modeling. 

The  availability  of  qualified  and  well-prepared  individuals  to  do  an  organization’s  process  performance 
modeling  is  another  important  measure  of  management  support.  The  relationship  in  Figure  50  is  only 
moderately  strong.  Recall  however  that  most  of  these  respondents  are  from  organizations  that  already 
have  achieved  high  maturity  status.  As  can  be  seen  by  the  relatively  narrow  columns  on  the  left-hand 


The  negatively  stated  items  were  reverse  scored  for  inclusion  into  the  composite  index  since  they  are  stated  nega- 
tiveiy  as  barriers  rather  than  facilitators  of  process  performance  modeling. 
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side  of  the  mosaic,  most  of  the  survey  respondents  report  that  the  necessary  expertise  most  frequently  is 

21 

available  when  it  is  needed. 


Figure  49:  Relationship  between  management  support  for  modeling  and  overall 
value  attributed  to  process  performance  models 


The  gamma  value  is  significant  though  attenuated  because  of  the  uneven  distribution  on  the  x  variable  and  the  unex¬ 
plained  dip  in  the  leftmost  column  of  respondents  who  report  that  their  organizations  have  achieved  mixed  value  or 
less  from  in  the  outcomes  of  their  process  performance  modeling  activities. 


47  I  CMU/SEI-2008-TR-024 


1.00- 


o 

CD 

eio.75- 

c/> 


0.50- 


□_ 

CL 

o 

o 

3 

cC 

> 


m  0.25- 

0 

> 

O 

0.00- 


Extremely  valuable 


Very  valuable 


Mixed  value  or  worse 


"cO 

c 

g 

‘c/>  w 
CO  w 
o  0 

o  “ 
O  o 


0 

E 

*c0  0 

I  B 


0 

3 

O' 

0 


I  I 
<  0 


Qualified  PPM  staff  available  (S2Q9) 


Gamma  =  .33;  p  <  .003; 
N  =  144 


Figure  50:  Relationship  between  process  performance  model  staff  availability 
and  overall  value  attributed  to  process  performance  models 


6.4  Stakeholder  Involvement  in  Setting  Measurement  and  Anaiysis  Goais  and  Objectives 

As  mentioned  in  Section  4. 1 ,  the  importance  of  stakeholder  involvement  is  widely  acknowledged  by 
measurement  experts.  The  inclusion  of  key  stakeholders  in  deciding  what  to  measure  and  why  to  do  so 
is  a  basic  notion  of  goal-driven  measurement  [Park  1996,  van  Solingen  1999].  It  also  is  cmcial  for  the 
CMMI  Measurement  and  Analysis  process  area,  particularly  in  specific  goal  1  and  specific  practice  1 . 1 
which  are  meant  to  ensure  that  measurement  objectives  and  activities  are  aligned  with  the  organiza¬ 
tional  unit’s  information  needs  and  objectives,  and  in  specific  practice  2.4  which  emphasizes  the  impor¬ 
tance  of  reporting  the  results  to  all  relevant  stakeholders.  SEI  technical  staff  and  others  have  observed 
incidents  where  insufficient  stakeholder  involvement  in  the  creation  of  process  performance  baselines 
and  models  has  seriously  jeopardized  their  productive  use,  and  there  is  empirical  evidence  that  the 
existence  of  such  support  can  increase  the  likelihood  of  the  success  of  measurement  programs  in  gen¬ 
eral  [Goldenson  1999,  Goldenson  2000,  Gopal  2002]. 

As  shown  in  Figure  5 1 ,  there  is  a  moderately  strong  positive  relationship  between  our  composite  meas¬ 
ure  of  stakeholder  involvement  and  the  respondents’  reports  of  the  effects  of  using  their  process  per¬ 
formance  models.  However,  the  apparent  effects  of  better  alignment  of  the  models  with  stakeholder 
input  are  seen  much  more  clearly  with  other  interim  y  factors,  as  shown  in  Figure  52  through  Figure  54. 
While  stakeholder  participation  appears  to  be  crucial  in  deciding  what  to  do  and  why,  its  effect  is  medi¬ 
ated  through  other  activities  that  are  more  closely  related  temporally  to  the  desired  outcomes. 


^  Participants  in  the  SEI’s  ongoing  workshop  series  on  measurement  and  analysis  in  high  maturity  organizations  often 
emphasize  that  domain  knowledge  is  even  more  important  than  expertise  in  measurement  and  analytic  methods 
[Stoddard  2008]. 
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Figure  51:  Relationship  between  stakeholder  involvement  and  overall  value 
attributed  to  process  performance  models 


As  seen  in  Figure  52,  there  is  a  moderately  strong  relationship  between  stakeholder  involvement  and 
the  use  of  process  performance  model  predictions  in  the  organizations’  status  and  milestone  reviews. 
Perhaps  more  importantly,  a  very  strong  relationship  can  been  seen  in  Figure  53  between  stakeholder 
involvement  and  the  emphasis  on  the  healthy  ingredients  that  the  organizations  put  in  their  process  per¬ 
formance  models.  The  relationship  shown  in  Figure  54  is  even  stronger  when  the  healthy-ingredient- 
based  purposes  for  which  the  models  are  used  is  the  interim  y  factor. 
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Figure  52:  Relationship  between  stakeholder  involvement  and  use  of  process  performance 
model  predictions  in  status  and  milestone  reviews 
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Figure  53:  Relationship  between  stakeholder  involvement  and  emphasis  on 
healthy  process  performance  model  ingredients 
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Figure  54:  Relationship  between  stakeholder  involvement  and  use  of  healthy 
process  performance  model  ingredients 


6.5  Measurement-Related  Training 

We  built  two  composite  measures  based  on  the  survey  respondents’  answers  to  the  two-question  series 
described  in  Figure  12  and  Figure  13  in  Section  4.2.  Recall  that  those  questions  refer  only  to  the  venues 
in  which  the  training  occurs  and  its  duration.  We  have  no  direct  measures  of  the  content  or  quality  of  that 
training.  We  also  do  not  know  how  many  courses  are  required  over  what  period  of  time.  Flence  there  may 
be  a  good  deal  of  statistical  noise  in  these  composite  variables.  Flowever,  the  results  do  suggest  that 
there  may  be  considerable  payoff  in  providing  more  and  better  training  and  coaching  for  management. 
The  situation  with  respect  to  training  for  modelers  is  less  clear  cut,  but  it  too  suggests  areas  where  train¬ 
ing  can  add  value. 

The  management-training  composite  is  based  on  the  responses  to  the  following:  “What  best  characterizes 
the  measurement-related  training  that  your  organization  requires  for  its  employees?”  It  sums  across  the 
answers  for  executive  and  senior  managers,  middle  managers,  and  project  managers. 

The  modeler-training  composite  is  based  on  the  responses  to  the  following:  “What  kind  of  specialized 
training,  if  any,  does  your  organization  require  for  its  employees  who  have  responsibilities  for  process 
performance  modeling?”  It  sums  across  the  answers  for  process  performance  model  builders  and  main- 
tainers  (e.g.,  Six  Sigma  black  belts  or  other  measurement  specialists),  coaches  and  mentors  who  assist  the 
model  builders  and  maintainers  (e.g.,  Six  Sigma  master  black  belts),  and  those  who  collect  and  manage 
the  baseline  data  (e.g.,  Six  Sigma  green  belts,  other  project  engineers,  or  EPG  members). 

^  The  training-related  questions  in  this  survey  were  chosen  largely  to  provide  simple  baselines  for  tracking  change  over 
time.  We  may  focus  on  training  in  more  detail  in  a  subsequent  survey. 
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Both  composite  measures  are  weighted  summed  indices.  The  component  items  were  weighted  based 
largely  on  the  duration  of  the  training.  Briefings  were  not  included  in  the  composite  connt  for  manage¬ 
ment,  as  of  course  there  were  instances  where  the  respondents  said  that  no  training  was  reqnired.  On- 
the-job,  online/self-paced,  and  tutorials  received  one  point.  One-  to  two-day  courses  received  two 
points.  One-  to  four-week  and  college  courses  received  three  points.  The  weights  were  the  same  for  the 
modeler  training,  except  college  training  received  4  points. 

The  relationship  between  our  measure  of  management  training  and  the  overall  value  attributed  by  the 
survey  respondents  to  the  outcomes  of  their  organizations’  process  performance  modeling  is  shown  in 
Figure  55.  The  relationship  is  moderately  strong.  So  too  is  the  relationship  in  Figure  56  where  the  ques¬ 
tion  about  the  use  of  process  performance  model  predictions  in  reviews  is  used  as  the  interim  y  vari¬ 
able.  There  also  are  moderately  strong  relationships  in  Figure  58  and  Figure  59,  where  the  questions 
about  the  managers’  level  of  understanding  of  process  performance  model  results  and  stakeholder  in¬ 
volvement  composite  measure  are  used  respectively  as  the  interim  y  variables. 


There  are  more  discordant  patterns  in  these  relationships  than  in  some  of  the  others  already  seen  in  the 
other  mosaics,  which  is  why  the  gamma  values  are  proportionally  lower.  That  is  not  snrprising  given 
the  limitations  of  the  management  training  measnre.  Notice  thongh  that  there  are  some  notable  differ¬ 
ences  when  one  compares  those  organizations  that  require  more  formal  training  courses  with  those  that 
do  not.  That  is  especially  apparent  when  the  nse  of  process  performance  model  predictions  in  reviews 
and  stakeholder  involvement  were  used  as  the  interim  y  variables.  Both  of  those  relationships  may  be 
more  directly  susceptible  to  management  control  than  the  other  two. 
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Figure  55:  Relationship  between  management  training  and  overall  value  attributed 

to  process  performance  models 


There  Is  no  statistically  significant  relationship  between  management  training  and  the  composite  measure  of  man¬ 
agement  support  that  Is  summarized  In  Figure  49.  That  Is  not  surprising  since  the  latter  can  be  affected  by  many 
things  other  than  training  per  se. 
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Figure  56:  Relationship  between  management  training  and  use  of  process 
performance  model  predictions  in  reviews 
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Figure  57:  Relationship  between  management  training  and  level  of  understanding 
of  process  performance  model  results  attributed  to  managers 
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Figure  58:  Relationship  between  management  training  and  stakeholder 
involvement  In  responding  organizations 


Perhaps  more  interestingly,  the  relationship  strengthens  when  the  interim  y  variable  is  the  extent  to 
which  the  organizations  emphasize  the  healthy  ingredients  of  process  performance  modeling  in  their 
own  modeling  efforts.  A  moderately  strong  to  strong  relationship  can  be  seen  in  Figure  59.  The  same  is 
so  in  Figure  60  where  use  of  diverse  statistical  methods  is  the  interim  y  variable.  These  too  may  be 

25 

more  under  management  control. 


All  the  other  x  variables  that  can  most  reasonably  be  expected  to  fall  more  directly  under  management  control  also 
exhibit  moderately  strong  relationships  when  they  are  considered  as  the  Interim  y  variable  for  management  training. 
These  Include  the  use  of  automated  support  (gamma  =  .35),  the  use  of  optimization  techniques  (gamma  =  .33),  and 
the  healthy-Ingredlent-based  purposes  for  which  the  models  are  used  (gamma  =  .27).  Of  course,  effects  on  training  It¬ 
self  may  be  reciprocal  over  time.  Improvements  to  an  organization’s  training  and  coaching  activities  may  be  more 
likely  when  using  the  models  has  been  perceived  as  valuable. 
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Figure  59:  Relationship  between  management  training  and  emphasis  on  healthy 
process  performance  model  ingredients 
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Figure  60:  Relationship  between  management  training  and  use  of  diverse  statistical  methods 

Modeler  training  appears  to  have  much  less  direct  effect.  Its  effects  probably  are  mediated  by  too  many 
other,  more  important  determinants  of  overall  value.  Moreover,  the  survey  respondents,  most  of  whom 
fill  management  roles  in  their  organizations,  may  not  have  a  particularly  good  sense  of  the  quality  of 
the  training  the  modelers  receive.  As  can  be  seen  in  Figure  61,  there  is  a  moderate  relationship  with  the 
overall  value  the  respondents  attribute  to  their  organizations’  process  performance  models  (gamma  = 
.29);  however,  the  relationships  between  the  management  training  measure  and  the  other  interim  y- 
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variables  are  almost  all  much  stronger.  There  are  weak  relationships  with  modeler  training  when  the 
use  of  automated  support  for  measurement  (gamma  =  .27),  the  use  of  exemplary  modeling  approaches 
(gamma  =  .24),  and  the  use  of  optimization  techniques  (gamma  =  .21)  are  used  as  interim  y  factors. 
Those  may  be  more  clearly  a  function  of  modeler  knowledge;  however,  no  other  measure  we  examined 
is  associated  in  a  statistically  significant  way  with  modeler  training. 
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Figure  61:  Relationship  between  modeler  training  and  overall  effect  attributed 

to  process  performance  modeling 


6.6  Technical  Challenge 

As  described  in  Section  5.2,  the  technical  challenges  sometimes  faced  by  projects  and  product  teams 
can  have  a  direct  effect  on  their  chances  of  completing  their  work  on  time,  on  budget,  and  with  the 
scope  of  the  deliverables  as  specified  contractually.  Different  degrees  of  technical  challenge  can  war¬ 
rant  different  degrees  of  process  capability  [Elm  2008,  Ferguson  2007].  Unlike  the  systems  engineering 
effectiveness  survey  just  cited,  the  results  in  Figure  62  show  no  such  direct  relationship.  Indeed  there  is 
essentially  no  association  between  our  measure  of  project  technical  challenge  and  the  value  of  the  out¬ 
comes  attributed  to  process  performance  modeling  by  the  survey  respondents. 

The  contrast  may  be  due  in  part  to  differences  in  the  measures  used  in  the  two  studies.  The  one  reported 
here  is  a  composite  of  the  respondents’  answers  to  the  question  series  that  is  summarized  in  Figure  34. 
Although  some  of  those  questions  relate  more  directly  to  organizational  context,  most  focus  heavily  on 
the  difficulty  inherent  in  certain  product  characteristics.  The  measure  used  in  the  other  survey  contains 
a  mix  focusing  on  constraints  posed  by  organizational  structure  and  interdependencies. 

Perhaps  more  likely,  the  contrast  in  results  may  also  be  due  to  differences  in  the  two  survey  samples. 
The  systems  engineering  survey  was  not  limited  to  high  maturity  organizations.^®  The  present  results 

The  systems  engineering  survey  was  based  on  iarge  defense  projects.  Aimost  60  percent  of  them  had  not  been  ap¬ 
praised  for  impiementing  processes  consistent  with  CMMi.  About  70  percent  of  the  rest  had  not  achieved  ievei  four  or 
five.  Aimost  aii  of  those  iower  maturity  projects  were  appraised  at  maturity  ievei  three. 
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may  simply  show  that  high  maturity  organizations  are  better  able  to  handle  harder  projects.  Moreover, 
most  of  the  responding  organizations  report  that  their  projects  typically  face  relatively  little  technical 
challenge  in  their  work.  As  shown  in  the  category  names  on  the  figure’s  x  axis,  a  very  large  proportion 
of  them  reported  that  the  statements  on  which  the  composite  measure  is  based  typically  described  their 
projects  to  less  than  a  moderate  extent. 


Figure  62:  Relationship  between  project  technical  challenge  and  overall  organizational  effect  attributed  to 

process  performance  modeling 

Yet  differences  in  project  technical  challenge  do  seem  to  matter  for  the  high  maturity  organizations 
described  in  this  report.  There  is  essentially  no  difference  in  the  relationship  between  the  technical 
challenge  composite  measure  and  the  value  of  the  effects  the  respondents  attribute  to  process  perform¬ 
ance  modeling,  as  shown  in  Figure  62.  However  the  relationships  of  other  measures  with  reported  or¬ 
ganizational  effects  do  in  fact  differ  consistently  as  a  function  of  the  extent  of  technical  challenge  typi¬ 
cally  faced  by  the  organizations’  projects. 

The  relationship  between  reported  organizational  effects  and  the  use  of  process  performance  model 
predictions  in  status  and  milestone  reviews  is  summarized  separately  in  Figure  63  for  organizations 
with  projects  that  face  relatively  little  technical  challenge.  The  same  relationship  is  summarized  in 
Figure  64  for  organizations  with  projects  that  face  relatively  more  technical  challenge.  The  relationship 
for  organizations  facing  lower  technical  challenge  is  exceptionally  high  for  survey  data  such  as  these. 
The  relationship  in  Figure  64  for  those  facing  relatively  more  technical  challenge  also  is  very  strong, 
although  it  is  somewhat  lower  than  the  overall  relationship  regardless  of  technical  challenge  that  is  de¬ 
scribed  in  Section  4.3. 


The  four  categories  of  project  technical  challenge  shown  in  Figure  62  are  dichotomized  here  because  of  the  small 
numbers  of  cases  and  resultant  fewer  degrees  of  freedom  for  statistical  analysis. 
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More  importantly  though,  the  organizations  with  more  difficult  projects  are  noticeably  more  likely  to 
report  more  valuable  effects  due  to  their  modeling  efforts  than  are  those  with  projects  that  face  fewer 
technical  challenges.  And  that  is  true  for  those  organizations  that  use  process  performance  model  re¬ 
sults  the  least  for  decision  making  in  their  status  and  milestone  reviews  as  well  as  for  those  who  use 
their  results  the  most. 


Expectations  of  what  constitutes  valuable  effects  may  be  lower  for  those  who  use  the  model  results  the 
least;  however,  the  fact  that  they  find  more  value  in  them  under  more  difficult  circumstances  is  particu¬ 
larly  telling,  as  shown  in  Figure  64.  Recall  that  all  of  the  organizations  surveyed  here  do  at  least  some 
process  performance  modeling.  Organizations  that  have  not  yet  begun  to  develop  such  capabilities 
might  also  achieve  better  outcomes  by  doing  so,  and  that  may  be  true  especially  for  those  that  face 
greater  technical  challenges. 
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Figure  63:  Relationship  between  use  of  process  performance  model  predictions  in  reviews  and  overall  effect 
attributed  to  process  performance  modeling,  with  lower  project  technical  challenge 
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Figure  64:  Relationship  between  use  of  process  performance  model  predictions  in  reviews  and  overall  effect 
attributed  to  process  performance  modeling,  with  higher  project  technical  challenge 

Somewhat  similar  results  can  be  seen  in  Figure  65  and  Figure  66  with  respect  to  the  emphasis  the  or¬ 
ganizations  put  on  the  healthy  ingredients  of  process  performance  models.  The  strength  of  association 
with  the  value  attributed  to  the  effects  of  the  modeling  is  similar  here  regardless  of  how  much  technical 
challenge  the  organizations  face.  Again,  the  respondents  attribute  more  valuable  effects  from  the  mod¬ 
eling  when  the  amount  of  technical  challenge  is  greater,  and  that  is  so  for  those  who  place  the  least  em¬ 
phasis  on  the  healthy  ingredients  as  well  as  those  who  place  the  most  emphasis  on  them.  Notice  also 
from  the  differing  widths  of  the  columns  in  these  two  figures  that  fewer  organizations  emphasize  the 
healthy  ingredients  when  project  technical  challenge  is  higher.  This  too  suggests  that  there  may  be  sub¬ 
stantial  opportunity  for  improvement  in  the  future  as  a  result  of  wider  and  more  effective  use  of  process 
performance  modeling. 
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Figure  65:  Relationship  between  emphasis  on  healthy  process  performance  model  ingredients  and  overall  effect 
attributed  to  process  performance  modeling,  with  lower  project  technical  challenge 


Figure  66:  Relationship  between  emphasis  on  healthy  process  performance  model  ingredients  and  overall 
effect  attributed  to  process  performance  modeling,  with  higher  project  technical  challenge 
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The  number  of  cases  on  which  both  of  these  comparisons  rely,  controlling  for  differences  in  project 
technical  challenge,  is  small.  However,  there  is  a  consistent,  similar  pattern  for  all  of  the  pertinent 
process  performance  modeling,  analytic  capability,  and  management  support  measures  that  we  exam¬ 
ined.  Organizations  with  relatively  more  difficult  projects  almost  always  reported  more  value  from 
their  modeling  than  did  those  with  less  difficult  projects,  and  that  is  especially  true  for  those  on  the 
lowest  end  of  the  x-variable  distributions.  The  distributions  compared  include  the  following: 

•  use  of  process  performance  model  predictions  in  reviews 

•  emphasis  on  healthy  process  performance  model  ingredients 

•  use  of  healthy  process  performance  model  ingredients 

•  exemplary  modeling  approaches 

•  diversity  of  process  performance  models:  product  quality  and  project  performance 

•  use  of  diverse  statistical  methods 

•  use  of  optimization  techniques 

•  use  of  automated  support  for  measurement  and  analysis  activities 

•  availability  of  qualified  process  performance  modeling  personnel 

•  management  support  (composite  measure) 

In  all  ten  comparisons,  the  responding  organizations  on  the  lowest  end  of  the  respective  x  variable  dis¬ 
tributions  were  more  likely  to  report  extremely  or  very  valuable  effects  of  their  process  performance 
modeling  when  their  reported  project  technical  challenge  was  higher.  Nine  of  the  ten  on  the  highest  end 
of  the  respective  x  variable  distributions  were  more  likely  to  report  extremely  valuable  effects  of  their 
process  performance  modeling  when  their  reported  project  technical  challenge  was  similarly  high.  The 
probability  of  either  proportion  occurring  simply  by  chance  is  quite  low,  even  for  such  a  small  number 

of  instances.  Based  on  a  simple  sign  test,  p  <.001  for  those  on  the  lowest  end  of  the  x  variable  distribu- 
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tions;  p  <  .02  for  those  on  the  highest  end. 

6.7  In  Summary:  A  Multivariable  Model 

It  is  particularly  difficult  to  tease  out  interpretations  of  cause  and  effect  when  several  variables  are  in¬ 
terrelated.  Such  is  the  case  in  this  study.  We  are  also  hampered  by  the  relatively  small  number  of  re¬ 
sponding  organizations.  We  are  able  to  make  what  appear  to  be  reasonable  conjectures  about  the  indi¬ 
rect  effects  of  stakeholder  involvement,  measurement-related  training,  and  project  technical  challenge 
in  Sections  6.4,  6.5,  and  6.6;  however  the  limited  numbers  of  cases  and  interrelated  joint  distributions 
prevent  further  explorations  about  those  conjectures  with  the  currently  available  data. 

However,  we  have  done  further  exploratory  data  analyses  to  describe  the  combined  impact  of  the  vari¬ 
ous  X  values  that  are  described  earlier  in  the  report.  To  what  extent  then  does  the  reported  organiza¬ 
tional  value  of  process  performance  modeling  change  as  a  function  of  variation  in  response  to  combina¬ 
tions  of  the  individual  questions  and  composite  measures?  The  interrelationships  are  quite  complex, 
often  with  mediating  effects,  so  it  is  difficult  to  describe  the  overall  relationship  simply. 


The  sign  test  is  a  simpie  inferentiai  statistic  based  on  the  binomiai  theorem.  A  ciear  description  may  be  found  in  the 
ciassic  reference  by  Siegei  and  Casteiian  [Siegei  1998].  A  convenient  sign  test  caicuiator  from  the  institute  of  Phonetic 
Sciences,  Amsterdam  is  availabie  at  http://www.fon.hum.uva.ni/Service/Statistics/Sign_Test.htmi. 
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We  began  by  examining  the  x  factors  that  are  most  strongly  associated  with  the  reported  outcomes  of 
process  performance  modeling.  Knowing  that  those  factors  are  often  quite  strongly  related  with  each 
other,  we  focused  on  several  combinations  of  x  factors  while  looking  for  as  simple  a  parsimonious 
model  as  possible.  After  using  several  statistical  methods  to  narrow  the  search,  we  used  multiple  logis¬ 
tic  regression  for  non-categorized  measures  to  settle  on  a  model  with  four  x  factors.  As  in  the  rest  of 
this  report,  the  single  question  about  overall  value  attributed  to  process  performance  modeling  is  used 
as  the  y  factor.^^ 

The  X  factors  in  the  best  model  we  found  include  the  following: 

•  use  of  process  performance  model  predictions  in  status  and  milestone  reviews 

•  diversity  of  process  performance  models:  product  quality  and  project  performance 

•  a  composite  measure  of  management  support  and  exemplary  modeling  approaches  that  facilitate 
process  performance  modeling^*^ 

•  emphasis  on  healthy  process  performance  model  ingredients 

Recall  that  the  strongest  bivariate  relationship  described  in  this  report  using  overall  outcome  as  the  y 
variable  is  the  one  with  use  of  process  performance  model  predictions  in  status  and  milestone  reviews 
as  the  single  x  variable  (gamma  =  .67).  The  strength  of  the  combined  relationship  achieved  with  the 
multivariable  model  increases  modestly  to  a  gamma  value  of  .7 1 . 


^  Clear  treatments  of  logistic  regression  methods  can  be  found  in  several  books  [Menard  2002,  Hosmer  2000,  Klein- 
baum  2002]. 

™  Combining  component  items  of  both  management  support  and  exemplary  modeling  approaches  into  a  single  compos¬ 
ite  measure  works  better  for  modeling  purposes  since  it  gets  at  both  conceptual  spaces  in  a  single  measure.  The 
combined  measure  is  based  only  on  items  that  are  phrased  positively  as  facilitators  rather  than  barriers. 
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7  Performance  Outcomes  of  Measurement  and  Analysis  Across 
Maturity  Levels 


As  part  an  ongoing  series  of  surveys  about  the  state  of  measurement  and  analysis  practice,  the  SEI  also 
conducted  a  companion  survey  in  2008  of  organizations  from  across  the  full  spectrum  of  CMMI-hased 
maturity.  This  section  contains  selected  results  from  that  survey.  They  highlight  the  potential  payoff  of 
measurement  and  analysis  in  lower  maturity  organizations. 

The  general  population  survey  had  a  response  rate  of  25  percent.  Similar  to  the  2007  survey  in  the  se¬ 
ries  [Goldenson  2008h],  there  is  evidence  that  measurement  and  analysis  capabilities  and  performance 
outcomes  typically  improve  considerably  as  the  survey  respondents’  organizations  move  up  in  maturity 
level.  All  of  the  respondents  in  the  results  shown  here  were  screened  to  ensure  that  they  use  measure¬ 
ment  and  analysis,  either  regularly  (62%)  or  at  least  occasionally  (20%). 

The  focus  in  this  report  is  on  three  process  variables:  CMMI  maturity  level,  the  use  of  product  and 
quality  measurement  results,  and  the  use  of  project  and  organizational  measurement  results.  Similar  to 
the  high  maturity  survey,  a  single  question  on  the  overall  value  of  measurement  and  analysis  is  used  as 
the  y  factor  in  this  report.  All  three  process  variables  vary  predictably  with  the  overall  value  of  meas¬ 
urement  and  analysis  that  the  survey  respondents  attribute  to  their  organizations. 

There  is  a  moderately  strong  to  strong  relationship  between  CMMI  maturity  and  the  respondents’  an¬ 
swers  about  how  valuable  measurement  and  analysis  has  been  to  their  organizations,  as  seen  in  Figure 
67.  CMMI-based  maturity  is  measured  by  a  single  question:  “To  the  best  of  your  knowledge,  what  is 
the  maturity  level  of  your  organization?”  The  results  follow  a  common  stair-step  pattern.  Over  twenty- 
five  percent  of  the  organizations  that  have  achieved  maturity  level  four  or  five  status  report  that  the 
results  have  been  extremely  valuable  for  their  organizations,  and  three-quarters  say  that  the  results  have 
been  very  valuable  or  extremely  valuable.  Decreasing  value  occurs  for  organizations  that  have  not  yet 
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achieved  high  maturity  status,  and  the  pattern  is  quite  the  opposite  at  maturity  level  one. 


As  with  the  high  maturity  survey,  we  aiso  examined  a  weighted,  summed  composite  measure  based  on  responses 
about  vaiue  added  by  measurement  and  anaiysis  to  project  performance,  product  quality,  and  tactical  and  strategic 
decisions.  The  results  are  similar  using  either  measure  as  the  y  factor.  Also  similar  to  the  high  maturity  survey,  the 
single  question  asks  “In  general,  how  valuable  has  measurement  and  analysis  been  to  your  organization?” 

A  relatively  small  number  of  the  survey  respondents  did  not  know  their  organizations’  maturity  levels.  They  are 
grouped  with  those  at  level  one  because  their  answers  to  the  other  survey  questions  are  essentially  the  same  as 
those  at  maturity  level  one.  Note  also  that  the  questionnaire  gave  the  respondents  an  opportunity  to  say  they  were 
“close  to”  the  next  level,  so  that  they  would  be  less  likely  to  err  by  overstating  their  true  status.  “Close  to”  replies  were 
grouped  with  the  next  lower  levels  since  maturity  level  is  a  discrete  concept. 
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Figure  67:  Relationship  between  maturity  level  and  overall  value  attributed  to  measurement  and  analysis 


Interestingly  enough,  the  two  other  process  variables  are  more  closely  associated  with  the  respondents’ 
reports  about  the  value  of  their  measurement  related  activities  to  their  respective  organizations  than  is 
maturity  level  alone.  That  is  not  surprising,  since  organizations  at  all  maturity  levels  do  vary  in  the  spe¬ 
cifics  of  what  they  measure  and  how  they  use  the  results. 


Use  of  product  and  quality  measurement  results  is  measured  by  a  weighted,  summed  index  based  on 
the  survey  respondents’  answers  about  how  often  a  series  of  several  kinds  of  product  and  quality  meas¬ 
urement  results  are  reported  in  their  organizations.  The  categories  include  the  following: 


•  product  requirements  or  architectures  (e.g.,  completion  of  customer  and  technical  requirements,  or 
features  delivered  as  planned) 

•  effort  applied  to  tasks  (e.g.,  productivity,  rework,  and  cost  of  quality  or  poor  quality) 

•  defect  density  (e.g.,  numbers  of  defects  identified  pre  and  post  release) 

•  defect  phase  containment  (i.e.,  early  detection  and  removal) 

•  quality  attributes  (e.g.,  maintainability,  interoperability,  portability,  usability,  reliability,  complex¬ 
ity,  criticality,  reusability,  or  durability) 

•  customer  satisfaction  (e.g.,  satisfaction  with  staff  responsiveness  or  fitness  for  use  of  the  delivered 
product) 

The  results  in  Figure  68  show  a  strong  degree  of  association  between  frequency  of  use  of  product  and 
quality  measurement  results  reported  by  the  respondents  and  the  overall  value  attributed  to  measure¬ 
ment  and  analysis  value  in  their  respective  organizations.  The  stair-step  pattern  is  considerably  more 
distinct  when  this  more  direct  measure  is  used  as  the  x  factor  than  was  tme  for  maturity  level  alone. 

The  organizations  that  report  the  most  use  of  such  measurement  results  are  even  more  likely  to  report 
that  their  measurement  and  analysis  activities  have  been  very  valuable  or  extremely  valuable  than  was 
true  for  high  maturity  status  alone,  yet  almost  90  percent  of  those  where  product  and  quality  measures 
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typically  are  used  less  than  occasionally  report  that  their  measurement  and  analysis  activities  have  been 
of  mixed  value  or  worse. 
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Figure  68:  Relationship  between  use  of  product  and  quality  measurement  results  and  overall 

value  attributed  to  measurement  and  analysis 


The  differences  with  respect  to  value  attributed  to  measurement  and  analysis  are  even  starker  for  fre¬ 
quency  of  use  of  project  and  organizational  measurement  results.  That  may  be  because  lower  maturity 
organizations  often  begin  the  measurement  and  analysis  activities  with  these  kinds  of  measures. 


Use  of  project  and  organizational  measurement  results  also  is  measured  by  a  weighted  summed  index. 
It  too  is  based  on  the  respondents’  answers  about  how  often  a  series  of  several  categories  of  measures 
were  reported  in  their  organizations.  The  following  categories  are  included  in  the  composite  measure: 


•  staff  adherence  to  development  work  processes 

•  cost  performance  or  other  measures  of  budget  predictability 

•  schedule  performance,  milestone  satisfaction,  or  other  measures  of  schedule  predictability 

•  accuracy  of  estimates  (e.g.,  effort,  cost,  or  schedule) 

•  product  cycle  time,  time  to  market,  or  delivery  rate 

•  business  growth  and  profitability  (e.g.,  market  share,  revenue  generated,  profits,  or 
return  on  investment) 


The  relationship  summarized  in  Figure  69  is  a  very  strong  one.  The  stair-step  differences  are  even  more 
pronounced  here.  While  fewer  organizations  used  project  and  organizational  performance  measurement 
results  less  than  occasionally  than  was  true  for  product  and  quality,  almost  none  of  those  organizations 
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claim  that  the  overall  value  of  their  measurement  and  analysis  activities  has  provided  more  than  mixed 
value. 
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Project  &  organizational  measurement  results  (S5Q1  composite) 


Figure  69:  Relationship  between  use  of  project  and  organizational  measurement  results  and  overall  value 

attributed  to  measurement  and  analysis 


As  was  shown  in  Figure  67,  other  things  being  equal,  higher  maturity  level  organizations  clearly  are 
more  likely  to  find  value  in  using  measurement  and  analysis  than  those  at  lower  maturity  levels.  How¬ 
ever,  maturity  level  itself  corresponds  with  the  performance  effects  largely  through  increased  use  of 
specific  measurement  results.  Some  of  the  relationships  between  maturity  level  and  overall  value  per¬ 
sist,  but  are  attenuated  (weaker)  when  examined  separately  by  frequency  of  use  of  measurement  re- 
suits.  The  use  of  measurement  results  continues  to  be  associated  with  overall  value  when  compared 
separately  by  different  maturity  levels.  Those  relationships  are  especially  strong  in  the  high  maturity 
organizations,  but  the  use  of  measurement  and  analyses  shows  value  even  in  maturity  level  one  organi¬ 
zations. 


The  relationships  for  frequency  of  use  of  project  and  organizational  performance  measurement  results 
are  shown  in  Figure  70  and  Figure  71.  Figure  70  shows  that  the  frequency  of  use  of  the  results  is  asso¬ 
ciated  with  the  range  of  value  attributed  to  measurement  and  analysis  in  maturity  level  four  and  five 
organizations.  The  same  comparison  is  made  for  the  maturity  level  one  organizations  in  Figure  71.^"^ 


In  the  interest  of  space,  those  differences  are  not  shown  here. 

The  comparisons  at  maturity  levels  two  and  three  are  somewhat  less  interesting,  but  the  relationships  persist  there  as 
well.  They  are  not  reported  here  in  the  interests  of  space.  Relatively  similar  maturity  level  differences  exist  for  fre¬ 
quency  of  use  of  product  and  quality  measurement  results,  especially  at  maturity  level  one.  However,  they  too  are 
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As  expected,  the  differences  in  Figure  70  are  quite  pronounced  at  maturity  levels  four  and  five,  and  the 
relationship  is  a  very  strong  one.  Notice  that  almost  none  of  the  higher  maturity  organizations  use  these 
kinds  of  measurement  results  less  than  frequently.  However  more  frequent  reporting  of  measurement 
results  appears  to  add  additional  value,  even  among  these  high  maturity  organizations. 
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Project  &  organizational  measurement  results  {S5Q1  composite) 


Figure  70:  ML  5  only  -  Relationship  between  use  of  project  /organizational  measurement 
results  and  overall  value  attributed  to  measurement  and  analysis 


omitted  from  this  report  in  the  interests  of  space  since  the  project  and  organizational  performance  measures  typically 
are  more  pertinent  for  lower  maturity  organizations. 
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More  frequent  use  and  reporting  of  measurement  results  also  can  add  value  for  lower  maturity  organi¬ 
zations,  even  for  those  at  level  one.  Not  surprisingly,  fewer  maturity  level  one  organizations  typically 
produce  and  report  measurement  results  on  a  frequent  basis;  however  the  overall  relationship  remains 
quite  strong.  More  importantly,  there  is  evidence  here  that  measurement -related  activities  can  begin  to 
add  noticeable  value  early-on,  when  organizations  first  begin  their  process  improvement  journeys. 


Figure  71:  ML1/DK  only  -  Relationship  between  use  of  project  /organizational  measurement 
results  and  overall  value  attributed  to  measurement  and  analysis 
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8  Summary  and  Conclusions 


All  of  the  high  maturity  organizations  in  the  survey  sample  reported  having  received  at  least  some  busi¬ 
ness  value  as  a  result  of  their  process  performance  modeling  activities  (see  Section  3).  They  most  often 
said  that  their  models  were  very  valuable  and  that  they  “have  obtained  much  useful  information  from 
them”  (52%);  eight  percent  say  that  they  were  extremely  valuable,  and  that  they  “couldn’t  do  [their] 
work  properly  without  them.”  Others  did  say  that  their  models  have  provided  mixed  value,  but  that  they 
“have  obtained  useful  information  on  occasion”  (38%),  and  a  very  few  reported  little  or  no  value  (2%). 
None  report  being  worse  off  as  a  result  of  their  process  performance  modeling  efforts.  More  impor¬ 
tantly,  differences  in  the  outcomes  of  the  modeling  reported  vary  consistently  as  a  function  of  how  well 
what  have  been  called  the  “healthy  ingredients”  of  process  performance  modeling  are  understood  in 
these  organizations  and  the  uses  to  which  the  results  have  been  put  (see  Section  6.2).  They  also  differ 
predictably  as  a  function  of  the  extent  to  which  a  varied  mix  of  several  statistical  and  analytic  methods 
is  used  in  the  modeling  (see  Section  6.2).  Similarly,  as  is  so  typical  for  process  improvement  in  general, 
the  overall  value  reported  varies  by  how  much  management  support  exists  for  process  performance 
modeling  in  these  organizations  (see  Section  6.3). 

While  much  of  the  relationship  appears  to  be  mediated  by  the  consequences  of  other,  interim  activities, 
the  reported  value  of  the  modeling  also  varies  by  the  extent  to  which  all  relevant  stakeholders  are  in¬ 
volved  in  deciding  what  to  model  and  the  reasons  why  the  modeling  should  be  done  (see  Section  6.4). 
The  same  is  true  for  the  extent  of  training  related  to  measurement  and  process  performance  modeling 
that  is  required  by  these  organizations  (see  Section  6.5).  The  business  value  of  their  process  perform¬ 
ance  modeling  efforts  is  not  affected  directly  by  differences  in  the  amount  of  technical  challenge  faced 
by  projects  in  the  responding  organizations;  however,  significantly  more  value  appears  to  be  added  un¬ 
der  more  difficult  circumstances.  That  is  true  in  spite  of  the  fact  that  fewer  resources  tend  to  be  allo¬ 
cated  to  process  performance  modeling  under  such  circumstances  (see  Section  6.6). 

As  seen  throughout  this  report,  many  of  the  statistical  relationships  just  mentioned  are  quite  strong  for 
survey  data  of  this  kind.  Altogether,  we  are  able  to  predict  over  70  percent  of  the  ranked  differences  in 
outcome  that  the  respondents  reported  about  the  business  value  of  their  process  performance  modeling 
activities  (see  Section  6.7).  Other  evidence  about  the  value  of  measurement  and  analysis  in  lower  ma¬ 
turity  organizations  is  discussed  briefly  in  Section  7. 

It  is  our  hope  that  this  first  survey  on  measurement  and  analysis  in  high  maturity  organizations  has  pro¬ 
vided  better  insight  than  was  previously  available  into  the  value  that  organizations  can  and  have 
achieved  as  outcomes  of  the  deployment  of  their  process  performance  baselines  and  models.  The  em¬ 
pirical  results  presented  here  may  provide  a  more  compelling  case  to  help  organizations  understand 
what  it  takes  to  be  truly  successful  using  process  performance  models  and  how  to  better  ensure  that  the 
deployment  of  such  models  succeeds.  Most  if  not  all  of  the  x  factors  that  we  have  analyzed  and  that 
seem  to  drive  successful  use  of  process  performance  modeling  are  controllable  by  management  and 
technical  actions.  The  results  may  better  arm  managers  and  deployment  champions  to  drive  business 
and  product  excellence  in  their  own  organizations  and  throughout  the  CMMI-based  process  improve¬ 
ment  community. 
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Appendix  A:  Questionnaire  for  the  Survey  of  High  Maturity 
Organizations 


This  appendix  contains  a  listing  of  all  of  the  survey  questions  that  have  forced-choice,  closed-ended 
answers.  It  is  annotated  with  the  number  of  responses  for  each  answer  and  the  percentage  of  the  total 
answers  that  each  answer  represents,  except  for  the  multiple  response  questions  which  present  only  the 
counts.  Response  counts  are  not  provided  for  questions  where  the  answers  vary  widely  and  a  concise 
summary  would  be  misleading.  We  have  also  included  data  tags  with  each  question  for  cross  reference 
(e.g.,  S1Q2).  The  free-form  textual  answers  to  selected  open-ended  questions  are  in  Appendix  C.  Fac¬ 
similes  of  the  invitations  and  reminder  letters  sent  to  the  survey  respondents  follow  the  question  and 

35 

answer  listing  in  this  appendix. 


We  also  sent  very  similar  personalized  reminders  to  those  who  had  begun  but  not  yet  completed  their  questionnaires. 
They  are  not  reproduced  here  in  the  interests  of  space. 
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Software  Engfneering  Instiiute 
Ca^K^^tellIHl 


The  State  of  Measurement  &  Analysis  2008: 

Survey  of  Applications  in  Support  of  High  Maturity  Practice 


1.  Which  of  the  following  best  describes  the  role  you  play  in  your  organization?  (S1Q2) 


Category 

Count 

% 

1  =  Executive  or  senior  manager 

92 

59 

2  =  Middle  manager  (e.g.,  program  or  product  line) 

23 

15 

3  =  Project  manager 

15 

10 

4  =  Project  engineer  or  other  technical  staff> 

5 

03 

5  =  Process  or  quality  engineer 

1 

01 

6  =  Measurement  specialist 

2 

01 

7  =  Other 

18 

12 

Total 

156 

2.  How  is  your  organization  best  described? 


Category 

Count 

% 

1  =  Commercial  off  the  shelf  (e.g.,  shrink-wrap  or  custom 
installation  of  enterprise  solutions  such  as  SAP  or  Oracle) 

7 

5 

2  =  Contracted  new  development,  (e.g.,  for  use  in  par¬ 
ticular  product  lines  or  other  novel  point  solutions=SlQ3) 

51 

33 

3  =  In-house  or  proprietary  development  or  maintenance 

25 

16 

4  =  Defense  contractor 

31 

20 

5  =  Other  government  contractor 

7 

5 

6  =  Department  of  Defense  or  military  organization 

3 

2 

7  =  Other  government  agency 

1 

1 

8  =  Other 

30 

19 

Total 

155 

3.  What  is  the  primary  focus  of  your  organization's  work? 


Category 

Count 

% 

1  =  Product  or  system  development 

97 

63 

2  =  Maintenance  or  sustainment 

19 

12 

3  =  Acquisition 

22 

14 

4  =  Service  provision 

16 

10 

Total 

154 
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4.  What  kinds  of  engineering  are  major  parts  of  your  organization's  work?  (Piease  seiect  as 
many  as  appiy)  [154  respondents] 


Category 

Count 

1  =  Software  engineering 

97 

2  =  Systems  engineering 

75 

3  =  Hardware  engineering 

31 

4  =  Design  engineering 

54 

5  =  Test  engineering 

75 

6  =  Other 

5 

5.  How  wouid  you  best  describe  your  invoivement  with  measurement  and  anaiysis? 


Category 

Count 

% 

1  =  1  am  a  provider  of  measurement-based  information 

6 

4 

2  =  1  am  a  user  (consumer)  of  measurement-based  information 

49 

31 

3  =  1  am  both  a  provider  and  user  (consumer)  of  measurement- 
based  information 

96 

62 

4  =  Other 

5 

3 

Total 

156 

6.  In  what  country  is  your  organization  primariiy  iocated? 


Category 

Count 

% 

1  =  United  States 

41 

26 

2  =  Canada 

4 

3 

3  =  China 

23 

15 

4  =  France 

0 

0 

5  =  Germany 

0 

0 

6  =  India 

46 

30 

7  =  Japan 

6 

4 

8  =  Netherlands 

1 

1 

9  =  United  Kingdom 

2 

1 

10  =  All  Others 

32 

21 

Total 

155 
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7.  Approximately  how  many  full-time  employees  in  your  organization  work  predominantly 
in  software,  hardware  or  systems  engineering  (e.g.,  development,  maintenance,  acquisi¬ 
tion  or  provision  of  related  services)? 


Category 

Count 

% 

1  =  25  or  fewer 

0 

0 

2  =  26-50 

2 

1 

3  =  51-75 

3 

2 

4  =  76-100 

3 

2 

5  =  101-200 

22 

14 

6  =  201-300 

23 

15 

7  =  301-500 

17 

11 

8  =  501-1000 

30 

19 

9  =  1001-2000 

18 

12 

10  =  More  than  2000 

38 

24 

Total 

156 

8.  To  the  best  of  your  knowledge,  what  is  the  current  maturity  level  of  your  organization? 
(Please  select  one.) 


Category 

Count 

% 

1  =  CMMI  Maturity  Level  3  or  lower 

6 

4 

2  =  Close  To  Maturity  Level  4 

6 

4 

3  =  CMMI  Maturity  Level  4 

25 

16 

4  =  CMMI  Maturity  Level  5 

118 

76 

5  =  Don't  know 

1 

1 

Total 

156 
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II.  Measurement  Related  Training  &  Staffing 

1.  What  best  characterizes  the  measurement  related  training  that  your  organization  re¬ 
quires  for  its  employees?  (Please  select  one  for  each) 


Executive  and  senior  managers 


Category 

Count 

% 

1  =  None 

6 

4 

2  =  On  the  job 

19 

13 

3  =  Online/self  paced 

9 

6 

4  =  Briefings 

75 

50 

5  =  Tutorials 

7 

5 

6  =  1-2  day  courses 

22 

15 

7  =  1-4  week  courses 

7 

5 

8  =  College  courses 

1 

1 

9  =  Don't  know 

3 

2 

10  =  Does  not  apply 

0 

0 

Total 

149 

Middle  managers  (e.g.,  program  or  product  line) 


Category 

Count 

% 

1  =  None 

1 

1 

2  =  On  the  job 

16 

11 

3  =  Online/self  paced 

10 

7 

4  =  Briefings 

22 

15 

5  =  Tutorials 

18 

12 

6  =  1-2  day  courses 

69 

46 

7  =  1-4  week  courses 

13 

9 

8  =  College  courses 

0 

0 

9  =  Don't  know 

0 

0 

10  =  Does  not  apply 

0 

0 

Total 

149 

Project  managers 


Category 

Count 

% 

1  =  None 

0 

0 

2  =  On  the  job 

9 

6 

3  =  Online/self  paced 

12 

8 

4  =  Briefings 

9 

6 

5  =  Tutorials 

15 

10 

6  =  1-2  day  courses 

76 

51 

7  =  1-4  week  courses 

28 

19 

8  =  College  courses 

0 

0 

9  =  Don't  know 

0 

0 

10  =  Does  not  apply 

0 

0 

Total 

149 
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Project  engineers  and  other  technical  staff 


Category 

Count 

% 

1  =  None 

0 

0 

2  =  On  the  job 

22 

15 

3  =  Online/self  paced 

11 

7 

4  =  Briefings 

12 

8 

5  =  Tutorials 

25 

17 

6  =  1-2  day  courses 

57 

38 

7  =  1-4  week  courses 

19 

13 

8  =  College  courses 

2 

1 

9  =  Don't  know 

0 

0 

10  =  Does  not  apply 

1 

1 

Total 

149 
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Process  or  quality  engineers 


Category 

Count 

% 

0  =  none  selected 

0 

0 

1  =  None 

1 

1 

2  =  On  the  job 

8 

5 

3  =  Online/self  paced 

10 

7 

4  =  Briefings 

2 

1 

5  =  Tutorials 

11 

7 

6  =  1-2  day  courses 

41 

28 

7  =  1-4  week  courses 

74 

50 

8  =  College  courses 

2 

1 

9  =  Don't  know 

0 

0 

10  =  Does  not  apply 

0 

0 

Total 

149 

2.  What  kind  of  specialized  training,  if  any,  does  your  organization  require  for  its  employ¬ 
ees  who  have  responsibilities  for  process  performance  modeling?  (Please  select 
one  for  each ) 

Process  performance  model  builders  &  maintainers  (e.g..  Six  Sigma  black  belts  or  other 
measurement  specialists) 


Category 

Count 

% 

1  =  None 

1 

1 

2  =  On  the  job 

7 

5 

3  =  Online/self 
paced 

4 

3 

4  =  Briefings 

2 

1 

5  =  Tutorials 

5 

3 

6  =  1-2  day  courses 

33 

22 

7  =  1-4  week 

courses 

79 

53 

8  =  College  courses 

13 

9 

9  =  Don't  know 

2 

1 

10  =  Does  not  apply 

3 

2 

Total 

149 
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Coaches  &  mentors  who  assist  the  model  builders  and  maintainers  (e.g.,  Six  Sigma  master 
black  belts) 


Category 

Count 

% 

1  =  None 

3 

2 

2  =  On  the  job 

8 

5 

3  =  Online/self  paced 

7 

5 

4  =  Briefings 

3 

2 

5  =  Tutorials 

7 

5 

6  =  1-2  day  courses 

40 

27 

7  =  1-4  week  courses 

57 

38 

8  =  College  courses 

11 

7 

9  =  Don't  know 

3 

2 

10  =  Does  not  apply 

10 

7 

Total 

149 

Those  who  collect  and  manage  the  baseline  data  (e.g.,  Six  Sigma  green  belts,  other  project 
engineers  or  EPG  members) 


Category 

Count 

% 

1  =  None 

0 

0 

2  =  On  the  job 

14 

9 

3  =  Online/self  paced 

9 

6 

4  =  Briefings 

5 

3 

5  =  Tutorials 

12 

8 

6  =  1-2  day  courses 

49 

33 

7  =  1-4  week  courses 

57 

38 

8  =  College  courses 

2 

1 

9  =  Don't  know 

0 

0 

10  =  Does  not  apply 

0 

0 

Total 

149 

Users  of  the  models 


Category 

Count 

% 

1  =  None 

1 

1 

2  =  On  the  job 

30 

20 

3  =  Online/self  paced 

8 

5 

4  =  Briefings 

21 

14 

5  =  Tutorials 

27 

18 

6  =  1-2  day  courses 

50 

34 

7  =  1-4  week  courses 

10 

7 

8  =  College  courses 

0 

0 

9  =  Don't  know 

0 

0 

10  =  Does  not  apply 

1 

1 

Total 

148 
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3.  In  what  ways  does  your  organization  ensure  that  its  process  performance  modei  buiiders 
and  maintainers  are  property  trained?  [156  respondents] 


Category 

Count 

Training  developed  and  delivered  internally  within  the  organization 

131 

Purchase  of  training  materials  that  are  developed  elsewhere  but  deliv¬ 
ered  internally 

45 

Contracts  with  external  training  services 

94 

Conferences  and  symposiums 

86 

We  hire  the  right  people  in  the  first  place 

51 

Other  (Please  describe  briefly) 

11 

4.  What,  if  any,  other  types  of  measurement  reiated  training  or  mentoring  does  your  or¬ 
ganization  provide?  (Piease  describe  briefiy) 

5.  Approximate^  how  many  peopie  in  your  organization  work  with  process  performance 
baseiines  and  modeis  -  as  part  of  their  expiicitiy  assigned  work  efforts?  (Piease  specify  a 
number  for  each  ...or  type  DK  if  you  don't  know) 


Those  who  collect  and  manage  the  baseline  data  (e.g.,  Six  Sigma 
green  belts,  other  project  engineers  or  EPG  members) _ 

Those  who  build  and  maintain  the  models  (e.g.,  Six  Sigma  black 
belts  or  other  measurement  specialists) _ 

Those  who  mentor  or  coach  the  model  builders  and  maintainers 
(e.g.,  Six  Sigma  master  black  belts) _ 

Those  who  use  the  model  results  to  inform  their  decision  making 


6.  Approximate^  how  many  peopie  buiid  or  maintain  the  modeis  and  baseiines  as  their 
primary  work  assignments?  (Piease  specify  a  number  for  each  ...or  type  DK  if 
you  don't  know) 


The  builders  and  maintainers 
Their  mentors  or  coaches 


7.  How  weii  do  the  peopie  who  create  your  organization's  process  performance  modeis  and 
baseiines  understand  the  intent  of  CM  MI?  (Piease  seiect  one  for  each) 

The  CMMI  definition  of  a  process  performance  modei 


Category 

Count 

% 

1  =  Extremely  well 

46 

31 

2  =  Very  well 

77 

52 

3  =  Moderately  well 

20 

14 

4  =  To  some  extent 

3 

2 

5  =  Hardly  at  all 

0 

0 

6  =  Don't  know 

2 

1 

Total 

148 
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The  CMMI  definition  of  a  process  performance  baseiine 


Category 

Count 

% 

1  =  Extremely  well 

56 

38 

2  =  Very  well 

72 

49 

3  =  Moderately  well 

17 

11 

4  =  To  some  extent 

1 

1 

5  =  Hardly  at  all 

0 

0 

6  =  Don't  know 

2 

1 

Total 

148 

The  circumstances  when  process  performance  baseiines  are  usefui 


Category 

Count 

% 

1  =  Extremely  well 

39 

26 

2  =  Very  well 

74 

50 

3  =  Moderately  well 

32 

22 

4  =  To  some  extent 

2 

1 

5  =  Hardly  at  all 

0 

0 

6  =  Don't  know 

1 

1 

Total 

148 

The  circumstances  when  process  performance  modeis  are  usefui 


Category 

Count 

% 

1  =  Extremely  well 

31 

21 

2  =  Very  well 

70 

47 

3  =  Moderately  well 

40 

27 

4  =  To  some  extent 

6 

4 

5  =  Hardly  at  all 

0 

0 

6  =  Don't  know 

1 

1 

Total 

148 

8.  How  weii  do  the  managers  in  your  organization  who  use  process  performance  modei  re¬ 
suits  understand  the  resuits  that  they  use?  (Piease  seiect  one.) 


Category 

Count 

% 

1  Extremely  well 

11 

7 

2  Very  well 

54 

36 

3  Moderately  well 

68 

46 

4  To  some  extent 

15 

10 

5  Hardly  at  all 

1 

1 

6  Don't  know 

0 

0 

Total 

149 
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9.  How  often  are  qualified,  well-prepared  people  available  to  work  on  process  performance 
modeling  in  your  organization  when  you  need  them  (i.e.,  people  with  sufficient  measure¬ 
ment  related  knowledge,  competence,  and  statistical  sophistication)?  (Please  select  one.) 


Category 

Count 

% 

1  =  Almost  always  (Greater  than  or  equal  to  80%) 

53 

36 

2  =  Frequently  (Greater  than  or  equal  to  60% 

48 

32 

3  =  About  half  of  the  time  (Greater  than  40%  but  less  than  60%) 

26 

17 

4  =  Occasionally  (Less  than  or  equal  to  40%) 

19 

13 

5  =  Rarely  if  ever  (Less  than  or  equal  to  20%) 

3 

2 

Total 

149 

10.  Does  your  organization  provide  promotion  or  financial  incentives  for  its  employees  that 
are  tied  to  the  deployment  and  adoption  of  measurement  and  analysis  (e.g.,  via  six  sigma 
belt  programs)?  (Please  select  as  many  as  apply)  [156  respondents] 


Category 

Count 

No 

72 

Yes  ...  for  executive  and  senior  managers 

38 

...  for  middle  managers  (e.q.,  program  or  product  line) 

48 

...  for  project  managers 

55 

...  for  project  engineers  and  other  technical  staff 

50 

...  for  others  (Please  describe  briefly) 

21 

Don't  know 

4 
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III.  Alignment,  Coordination  &  Infrastructure 


1.  How  would  you  characterize  the  involvement  of  various  potential  stakeholders  in  setting 
goals  and  deciding  on  plans  of  action  for  measurement  and  analysis  in  your  organization? 
(Please  select  one  for  each) 

Customers 


Category 

Count 

% 

1  =  Extensive 

12 

8 

2  =  Substantial 

19 

13 

3  =  Moderate 

48 

33 

4  =  Limited 

37 

25 

5  =  Little  if  any 

14 

10 

6  =  Don't  know 

3 

2 

7  =  Does  not  apply 

13 

9 

Total 

146 
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Executive  and  senior  managers 


Category 

Count 

% 

1  =  Extensive 

48 

33 

2  =  Substantial 

52 

36 

3  =  Moderate 

37 

26 

4  =  Limited 

6 

4 

5  =  Little  if  any 

1 

1 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

0 

0 

Total 

144 

Middle  managers  (e.g.,  program  or  product  line) 


Category 

Count 

% 

1  =  Extensive 

40 

28 

2  =  Substantial 

61 

42 

3  =  Moderate 

37 

26 

4  =  Limited 

5 

3 

5  =  Little  if  any 

2 

1 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

0 

0 

Total 

144 

Project  managers 


Category 

Count 

% 

1  =  Extensive 

45 

31 

2  =  Substantial 

58 

40 

3  =  Moderate 

30 

21 

4  =  Limited 

9 

6 

5  =  Little  if  any 

2 

1 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

1 

1 

Total 

145 

Project  engineers  and  other  technicai  staff 


Category 

Count 

% 

1  =  Extensive 

5 

3 

2  =  Substantial 

31 

21 

3  =  Moderate 

53 

37 

4  =  Limited 

44 

30 

5  =  Little  if  any 

7 

5 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

5 

3 

Total 

145 
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Process  and  quality  engineers 


Category 

Count 

% 

1  =  Extensive 

63 

43 

2  =  Substantial 

55 

38 

3  =  Moderate 

19 

13 

4  =  Limited 

6 

4 

5  =  Little  if  any 

1 

1 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

1 

1 

Total 

145 

Measurement  specialists 


Category 

Count 

% 

1  =  Extensive 

68 

47 

2  =  Substantial 

44 

30 

3  =  Moderate 

19 

13 

4  =  Limited 

7 

5 

5  =  Little  if  any 

2 

1 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

5 

3 

Total 

145 

Others  (Please  describe  briefly)  name=S3Ql_9_txt 


Category 

Count 

% 

1  =  Extensive 

6 

25 

2  =  Substantial 

1 

4 

3  =  Moderate 

1 

4 

4  =  Limited 

0 

0 

5  =  Little  if  any 

0 

0 

6  =  Don't  know 

2 

8 

7  =  Does  not  apply 

14 

58 

Total 

24 

2.  Which  of  the  following  best  describes  how  work  on  measurement  and  analysis  is  staffed 
in  your  organization?  (Please  select  one.) 


Category 

Count 

% 

1  =  An  organization-wide,  division  or  similar  corporate  support  group 
(e.g.,  an  engineering  process,  quality  assurance  or  measurement  group) 

109 

76 

2  =  Separate  groups  or  individuals  in  different  projects  or  other  organ¬ 
izational  units  (e.g.,  project,  product  team  or  similar  work  groups 

22 

15 

3  =  A  few  key  people  (or  one  person)  in  the  organization  who  are  mea¬ 
surement  experts 

8 

6 

4  =  Other  (Please  describe  briefly) 

5 

3 

Total 

144 
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3.  How  much  automated  support  is  available  for  measurement  related  activities  in  your  or¬ 
ganization?  (Please  select  one  for  each.) 

Data  collection  (e.g.,  on-line  forms  with  "tickler"  reminders,  time  stamped  activity  logs, 
static  or  dynamic  analyses  of  call  graphs  or  run-time  behavior) 


Category 

Count 

% 

1  =  Extensive 

41 

28 

2  =  Substantial 

45 

31 

3  =  Moderate 

30 

21 

4  =  Limited 

21 

14 

5  =  Little  if  any 

6 

4 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

2 

1 

Total 

145 

Commercial  work  flow  automation  that  supports  data  collection 


Category 

Count 

% 

1  =  Extensive 

22 

15 

2  =  Substantial 

27 

19 

3  =  Moderate 

31 

21 

4  =  Limited 

25 

17 

5  =  Little  if  any 

20 

14 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

20 

14 

Total 

145 

Data  management  (e.g.,  relational  or  distributed  database  packages,  open  database  con¬ 
nectivity,  tools  for  data  integrity,  verification,  or  validation) 


Category 

Count 

% 

1  =  Extensive 

37 

26 

2  =  Substantial 

45 

31 

3  =  Moderate 

37 

26 

4  =  Limited 

14 

10 

5  =  Little  if  any 

9 

6 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

3 

2 

Total 

145 
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Spreadsheet  add-ons  for  basic  statistical  analysis 


Category 

Count 

% 

1  =  Extensive 

47 

33 

2  =  Substantial 

48 

34 

3  =  Moderate 

24 

17 

4  =  Limited 

19 

13 

5  =  Little  if  any 

3 

2 

6  =  Don't  know 

2 

1 

7  =  Does  not  apply 

0 

0 

Total 

143 

Commercial  statistical  packages  that  support  more  advanced  analyses 


Category 

Count 

% 

1  =  Extensive 

23 

16 

2  =  Substantial 

30 

21 

3  =  Moderate 

28 

19 

4  =  Limited 

23 

16 

5  =  Little  if  any 

19 

13 

6  =  Don't  know 

2 

1 

7  =  Does  not  apply 

19 

13 

Total 

144 

Customized  spreadsheets  for  routine  analyses  (e.g.,  for  defect  phase  containment) 


Category 

Count 

% 

1  =  Extensive 

43 

30 

2  =  Substantial 

54 

38 

3  =  Moderate 

28 

19 

4  =  Limited 

10 

7 

5  =  Little  if  any 

5 

3 

6  =  Don't  know 

1 

1 

7  =  Does  not  apply 

3 

2 

Total 

144 
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Commercial  software  for  report  preparation  (e.g.,  graphing  packages  or  other  presentation 
quality  results) 


Category 

Count 

% 

1  =  Extensive 

17 

12 

2  =  Substantial 

27 

19 

3  =  Moderate 

32 

22 

4  =  Limited 

28 

19 

5  =  Little  if  any 

18 

12 

6  =  Don't  know 

2 

1 

7  =  Does  not  apply 

21 

14 

Total 

145 

Other  (Please  describe  briefly) 


Category 

Count 

% 

1  =  Extensive 

8 

27 

2  =  Substantial 

4 

13 

3  =  Moderate 

2 

7 

4  =  Limited 

0 

0 

5  =  Little  if  any 

0 

0 

6  =  Don't  know 

2 

7 

7  =  Does  not  apply 

14 

47 

Total 

30 

4.  How  often  does  your  organization  do  the  following  with  the  data  it  collects?  (Please 
check  one  for  each) 

Check  for  out  of  range  or  other  illegal  values  in  the  recorded  data 


Category 

Count 

% 

1  =  Almost  always 

86 

60 

2  =  Frequently 

45 

31 

3  =  About  half  the  time 

7 

5 

4  =  Occasionally 

6 

4 

5  =  Rarely  if  ever 

0 

0 

6  =  Don't  know 

0 

0 

Total 

144 

Evaluate  the  number  and  distribution  of  missing  data 


Category 

Count 

% 

1  =  Almost  always 

57 

40 

2  =  Frequently 

59 

41 

3  =  About  half  the  time 

7 

5 

4  =  Occasionally 

14 

10 

5  =  Rarely  if  ever 

5 

4 

6  =  Don't  know 

1 

1 

Total 

143 
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Ensure  that  missing  data  are  not  inadvertentiy  treated  as  zero  vaiues 


Category 

Count 

% 

1  =  Almost  always 

78 

54 

2  =  Frequently 

40 

28 

3  =  About  half  the  time 

9 

6 

4  =  Occasionally 

6 

4 

5  =  Rarely  if  ever 

7 

5 

6  =  Don't  know 

4 

3 

Total 

144 

Check  for  precision  and  accuracy  of  the  data 


Category 

Count 

% 

1  =  Almost  always 

62 

43 

2  =  Frequently 

53 

37 

3  =  About  half  the  time 

13 

9 

4  =  Occasionally 

15 

10 

5  =  Rarely  if  ever 

0 

0 

6  =  Don't  know 

0 

0 

Total 

143 

Estimate  measurement  error  statisticaiiy 


Category 

Count 

% 

1  =  Almost  always 

20 

14 

2  =  Frequently 

38 

27 

3  =  About  half  the  time 

14 

10 

4  =  Occasionally 

40 

28 

5  =  Rarely  if  ever 

27 

19 

6  =  Don't  know 

4 

3 

Total 

143 

Check  for  inconsistent  interpretations  of  measurement  definitions 


Category 

Count 

% 

1  =  Almost  always 

41 

29 

2  =  Frequently 

67 

47 

3  =  About  half  the  time 

12 

8 

4  =  Occasionally 

19 

13 

5  =  Rarely  if  ever 

2 

1 

6  =  Don't  know 

2 

1 

Total 

143 
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Check  for  consistency/reliability  of  measurement  results  and  procedures  across  time  and 
reporting  units 


Category 

Count 

% 

1  =  Almost  always 

38 

27 

2  =  Frequently 

75 

52 

3  =  About  half  the  time 

8 

6 

4  =  Occasionally 

16 

11 

5  =  Rarely  if  ever 

4 

3 

6  =  Don't  know 

2 

1 

Total 

143 

Check  for  consistency  of  classification  decisions  based  on  the  same  information,  (otherwise 
known  as  in  ter- coder  reliability) 


Category 

Count 

% 

1  =  Almost  always 

23 

16 

2  =  Frequently 

54 

38 

3  =  About  half  the  time 

11 

8 

4  =  Occasionally 

27 

19 

5  =  Rarely  if  ever 

15 

11 

6  =  Don't  know 

12 

8 

Total 

142 

Analyze  &  address  the  reasons  for  unusual  patterns  in  the  data  distributions,  e.g.,  outliers, 
skewness,  or  other  aspects  of  non  normal  distributions 


Category 

Count 

% 

1  =  Almost  always 

61 

43 

2  =  Frequently 

55 

39 

3  =  About  half  the  time 

10 

7 

4  =  Occasionally 

13 

9 

5  =  Rarely  if  ever 

2 

1 

6  =  Don't  know 

1 

1 

Total 

142 

Analyze  &  address  the  reasons  for  unusual  or  unanticipated  relationships  between  two  or 
more  measures 


Category 

Count 

% 

1  =  Almost  always 

44 

31 

2  =  Frequently 

50 

35 

3  =  About  half  the  time 

15 

11 

4  =  Occasionally 

25 

18 

5  =  Rarely  if  ever 

6 

4 

6  =  Don't  know 

2 

1 

Total 

142 
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Automate  data  quality/integrity  checks  for  ease  of  collecting  consistent  data 


Category 

Count 

% 

1  =  Almost  always 

29 

21 

2  =  Frequently 

48 

34 

3  =  About  half  the  time 

14 

10 

4  =  Occasionally 

28 

20 

5  =  Rarely  if  ever 

19 

13 

6  =  Don't  know 

3 

2 

Total 

141 

Other  (Please  describe  briefly) 


Category 

Count 

% 

1  =  Almost  always 

5 

23 

2  =  Frequently 

3 

14 

3  =  About  half  the  time 

0 

0 

4  =  Occasionally 

0 

0 

5  =  Rarely  if  ever 

1 

5 

6  =  Don't  know 

13 

59 

Total 

22 
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IV.  Use  of  Process  Performance  Models  &  Baselines 


1.  For  what  operational  purposes  are  models  and  baselines  routinely  used  in  your  project 
and  organizational  product  development,  maintenance  or  acquisition  activities?  (Please  se¬ 
lect  as  many  as  apply  ...  or  be  sure  to  check  'None  of  the  above'  if  appropriate.)  [156  re¬ 
spondents] 


Category 

Count 

Defining  the  organization's  standard  processes  (e.g.,  implementing  the 
entire  process  asset  library  in  a  simulation  model) 

80 

Composing  projects' defined  processes  (e.g.,  modeling  trade-offs  in 
cost,  schedule  and  quality  to  select  among  alternative  subprocesses  or 
compositions  for  a  given  project) 

100 

Risk  management 

82 

Project  planning 

128 

Project  monitoring  and  corrective  actions 

131 

Identifying  opportunities  for  process  or  technology  improvement 

114 

Evaluating  process  or  technology  improvements 

103 

Other  (Please  describe  briefly) 

7 

None  of  the  above 

0 

Don't  know 

0 

2.  Where  else  in  the  organization  are  process  performance  models  and  baselines  used? 
(Please  select  as  many  as  apply  ...  or  be  sure  to  check  'None  of  the  above'  if  appropriate.) 
[152  respondents] 


Category 

Count 

Corporate  planning  and  strategy 

51 

Portfolio  planning 

22 

Responses  to  requests  for  proposals  or  tender  offers  (e.g.,  cost  and 
achievability  modeling) 

75 

Marketing 

26 

Technology  R&D 

39 

Business  operations 

63 

Supply  chain  management 

10 

Human  resources  management 

36 

Other  (Please  describe  briefly) 

5 

None  of  the  above 

20 

Don't  know 

11 
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3.  Which  of  the  following  product  quality  and  project  performance  outcomes  are  routinely 
predicted  with  process  performance  models  in  your  organization?  (Please  select  as  many  as 
apply  ...  or  be  sure  to  check  'None  of  the  above'  if  appropriate.)  [156  respondents] 


Category 

Count 

Delivered  defects 

127 

Type  or  severity  of  defects 

68 

Product  quality  attributes  (e.g.,  mean  time  to  failure,  design  complex¬ 
ity,  maintainability,  interoperability,  portability,  usability,  reliability, 
complexity,  reusability  or  durability) 

58 

Quality  of  services  provided  (e.g.,  IT  ticket  resolution  time) 

66 

Cost  and  schedule  duration 

119 

Work  product  size 

53 

Accuracy  of  estimates  (e.g.,  cost,  schedule,  product  size  or  effort) 

98 

ROI  of  process  improvement  or  related  financial  performance 

35 

Customer  satisfaction 

58 

Other  (Please  describe  briefly) 

3 

None  of  the  above 

1 

Don't  know 

0 

4.  Which  of  the  following  (often  interim)  process  performance  outcomes  are  routinely  pre¬ 
dicted  with  process  performance  models  in  your  organization?  (Please  select  as  many  as 
apply  ...  or  be  sure  to  check  'None  of  the  above'  if  appropriate)  [156  respondents] 


Category 

Count 

Escaped  defects  (e.g.,  as  predicted  by  defect  phase  containment  models) 

98 

Cost  of  quality  and  poor  quality  (e.g.,  rework) 

69 

Estimates  at  completion  (i.e.,  performed  periodically  throughout  the  pro¬ 
ject) 

108 

Requirements  volatility  or  qrowth 

41 

Effectiveness  or  efficiency  of  inspection  or  test  coverage 

83 

Practitioner  adherence  to  defined  processes 

38 

Other  (Please  describe  briefly) 

5 

None  of  the  above 

4 

Don't  know 

1 

5.  Which  of  the  following  processes  and  activities  are  routinely  modeled  in  your  organiza¬ 
tion?  (Please  select  as  many  as  apply  ...  or  be  sure  to  check  'None  of  the  above'  if  appro- 
priate)[156  respondents] 


Category 

Count 

Project  planning  and  estimation 

121 

Requirements  engineering 

79 

Product  architecture 

35 

Software  design  and  coding 

101 

Process  documentation 

48 

Quality  control  processes 

108 
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Systems  engineering  processes 

47 

Hardware  engineering  processes 

18 

Acquisition  or  supplier  processes 

9 

Other  (Please  describe  briefly) 

2 

None  of  the  above 

1 

Don't  know 

0 

6.  How  much  emphasis  does  your  organization  piace  upon  the  foiiowing  in  its  process  per¬ 
formance  modeiing?  (Piease  seiect  one  for  each.) 

Accounting  for  uncertainty  and  variabiiity  in  predictive  factors  and  predicted  outcomes 


Category 

Count 

% 

1  =  Extensive 

25 

18 

2  =  Substantial 

44 

31 

3  =  Moderate 

36 

25 

4  =  Limited 

20 

14 

5  =  Little  if  any 

10 

7 

6  =  Don't  know 

3 

2 

7  =  Does  not  apply 

4 

3 

Total 

142 

Factors  that  are  under  management  or  technicai  controi 


Category 

Count 

% 

1  =  Extensive 

26 

19 

2  =  Substantial 

60 

43 

3  =  Moderate 

34 

24 

4  =  Limited 

11 

8 

5  =  Little  if  any 

4 

3 

6  =  Don't  know 

3 

2 

7  =  Does  not  apply 

2 

1 

Total 

140 

Other  product,  contractuai  or  organizationai  characteristics,  resources  or  constraints 


Category 

Count 

% 

1  =  Extensive 

13 

9 

2  =  Substantial 

36 

25 

3  =  Moderate 

39 

27 

4  =  Limited 

21 

15 

5  =  Little  if  any 

20 

14 

6  =  Don't  know 

4 

3 

7  =  Does  not  apply 

9 

6 

Total 

142 
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Segmenting  or  otherwise  accounting  for  uncontroiiabie  factors 


Category 

Count 

% 

1  =  Extensive 

10 

7 

2  =  Substantial 

25 

18 

3  =  Moderate 

35 

25 

4  =  Limited 

32 

23 

5  =  Little  if  any 

20 

14 

6  =  Don't  know 

8 

6 

7  =  Does  not  apply 

12 

8 

Total 

142 

Factors  that  are  tied  to  detaiied  subprocesses 


Category 

Count 

% 

1  =  Extensive 

22 

15 

2  =  Substantial 

52 

36 

3  =  Moderate 

39 

27 

4  =  Limited 

13 

9 

5  =  Little  if  any 

11 

8 

6  =  Don't  know 

3 

2 

7  =  Does  not  apply 

3 

2 

Total 

143 

Factors  that  are  tied  to  iarger,  more  broadiy  defined  organizationai  processes 


Category 

Count 

% 

1  =  Extensive 

17 

12 

2  =  Substantial 

45 

32 

3  =  Moderate 

46 

32 

4  =  Limited 

17 

12 

5  =  Little  if  any 

10 

7 

6  =  Don't  know 

5 

4 

7  =  Does  not  apply 

2 

1 

Total 

142 

Other  (Piease  describe  briefiy) 


Category 

Count 

% 

1  =  Extensive 

0 

0 

2  =  Substantial 

0 

0 

3  =  Moderate 

4 

17 

4  =  Limited 

0 

0 

5  =  Little  if  any 

0 

0 

6  =  Don't  know 

4 

17 

7  =  Does  not  apply 

15 

65 

Total 

23 

7.  To  what  degree  are  your  organization's  process  performance  modeis  used  for  the  foiiow- 
ing  purposes?  (Piease  seiect  one  for  each.) 
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Predict  final  project  outcomes 


Category 

Count 

% 

1  =  Extensive 

45 

32 

2  =  Substantial 

52 

37 

3  =  Moderate 

33 

23 

4  =  Limited 

11 

8 

5  =  Little  if  any 

0 

0 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

0 

0 

Total 

141 

Predict  interim  outcomes  during  project  execution  (e.g.,  connecting  "upstream"  with  "down¬ 
stream  "  activities) 


Category 

Count 

% 

1  =  Extensive 

39 

27 

2  =  Substantial 

47 

33 

3  =  Moderate 

35 

25 

4  =  Limited 

16 

11 

5  =  Little  if  any 

2 

1 

6  =  Don't  know 

1 

1 

7  =  Does  not  apply 

2 

1 

Total 

142 

Model  the  variation  of  factors  and  understand  the  predicted  range  or  variation  of  the  pre¬ 
dicted  outcomes 


Category 

Count 

% 

1  =  Extensive 

24 

17 

2  =  Substantial 

43 

30 

3  =  Moderate 

36 

25 

4  =  Limited 

27 

19 

5  =  Little  if  any 

12 

8 

6  =  Don't  know 

1 

1 

7  =  Does  not  apply 

1 

1 

Total 

144 

Enable  "what-if"  analysis  for  project  planning,  dynamic  re-planning  and  problem  resolution 
during  project  execution  _ _ _ 


Category 

Count 

% 

1  =  Extensive 

23 

16 

2  =  Substantial 

42 

30 

3  =  Moderate 

37 

26 

4  =  Limited 

27 

19 

5  =  Little  if  any 

10 

7 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

3 

2 

Total 

142 
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Enable  projects  to  achieve  mid-course  corrections  to  ensure  project  success 


Category 

Count 

% 

1  =  Extensive 

34 

24 

2  =  Substantial 

50 

35 

3  =  Moderate 

38 

27 

4  =  Limited 

18 

13 

5  =  Little  if  any 

3 

2 

6  =  Don't  know 

0 

0 

7  =  Does  not  apply 

0 

0 

Total 

143 

Other  (Please  describe  briefly) 


Category 

Count 

% 

1  =  Extensive 

1 

5 

2  =  Substantial 

0 

0 

3  =  Moderate 

2 

11 

4  =  Limited 

0 

0 

5  =  Little  if  any 

0 

0 

6  =  Don't  know 

5 

26 

7  =  Does  not  apply 

11 

55 

Total 

19 
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V.  other  Analytic  Methods  &  Techniques 


1.  To  what  extent  are  the  following  statistical  methods  used  in  your  organization's  process 
performance  modeling?  (Please  select  one  for  each.) 

Regression  analysis  predicting  continuous  outcomes  (e.g.,  bivariate  or  multivariate  linear 
regression  or  non-linear  regression) 


Category 

Count 

% 

1  =  Extensive 

29 

21 

2  =  Substantial 

27 

19 

3  =  Moderate 

37 

26 

4  =  Limited 

27 

19 

5  =  Little  if  any 

10 

7 

6  =  Don't  know 

4 

3 

7  =  Does  not  apply 

7 

5 

Total 

141 

Regression  analysis  predicting  categorical  outcomes  (e.g.,  logistic  regression  or  loglinear 
models) 


Category 

Count 

% 

1  =  Extensive 

16 

11 

2  =  Substantial 

9 

6 

3  =  Moderate 

27 

19 

4  =  Limited 

34 

24 

5  =  Little  if  any 

21 

15 

6  =  Don't  know 

13 

9 

7  =  Does  not  apply 

20 

14 

Total 

140 

Analysis  of  variance  (e.g.,  ANOVA,  ANCOVA  or  MANOVA) 


Category 

Count 

% 

1  =  Extensive 

16 

12 

2  =  Substantial 

26 

19 

3  =  Moderate 

28 

20 

4  =  Limited 

27 

19 

5  =  Little  if  any 

23 

17 

6  =  Don't  know 

11 

8 

7  =  Does  not  apply 

8 

6 

Total 

139 
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Attribute  SPC  charts  (e.g.,  c,  u,  p,  or  np) 


Category 

Count 

% 

1  =  Extensive 

27 

19 

2  =  Substantial 

29 

20 

3  =  Moderate 

25 

17 

4  =  Limited 

26 

18 

5  =  Little  if  any 

15 

10 

6  =  Don't  know 

10 

7 

7  =  Does  not  apply 

11 

8 

Total 

143 

Individual  point  SPC  charts  (e.g.,  ImR  or  XmR) 


Category 

Count 

% 

1  =  Extensive 

43 

30 

2  =  Substantial 

45 

32 

3  =  Moderate 

27 

19 

4  =  Limited 

14 

10 

5  =  Little  if  any 

3 

2 

6  =  Don't  know 

6 

4 

7  =  Does  not  apply 

4 

3 

Total 

142 

Continuous  SPC  charts  (e.g.,  XbarR  or  XbarS) 


Category 

Count 

% 

1  =  Extensive 

31 

22 

2  =  Substantial 

30 

21 

3  =  Moderate 

26 

18 

4  =  Limited 

21 

15 

5  =  Little  if  any 

15 

11 

6  =  Don't  know 

7 

5 

7  =  Does  not  apply 

12 

8 

Total 

142 
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Design  of  experiments 


Category 

Count 

% 

1  =  Extensive 

4 

3 

2  =  Substantial 

7 

5 

3  =  Moderate 

20 

14 

4  =  Limited 

35 

25 

5  =  Little  if  any 

36 

26 

6  =  Don't  know 

10 

7 

7  =  Does  not  apply 

27 

19 

Total 

139 

Other  (Piease  describe  briefiy) 


Category 

Count 

% 

1  =  Extensive 

5 

17 

2  =  Substantial 

1 

3 

3  =  Moderate 

1 

3 

4  =  Limited 

1 

3 

5  =  Little  if  any 

0 

0 

6  =  Don't  know 

4 

13 

7  =  Does  not  apply 

18 

60 

Total 

30 

2.  Which  of  the  foiiowing  visuai  dispiay  techniques  are  used  to  communicate  the  resuits  of 
your  organization's  anaiyses  of  process  performance  baseiines?  (Piease  seiect  as  many  as 
appiy  ...  or  be  sure  to  check  'None  of  the  above'  if  appropriate)  [156  respondents] 


Category 

Count 

Box  plots 

80 

Histograms 

124 

Scatter  plots  or  multivariate  charting 

114 

Pareto  charts,  pie  charts  or  bar  charts 

129 

Mosaic  charts  for  categorical  data 

12 

Other  (Please  describe  briefly) 

11 

None  of  the  above 

0 

Don't  know 

0 
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3.  Which  of  the  following  other  optimization  approaches  are  used  in  your  organization's 
process  performance  modeling?  (Please  select  as  many  as  apply  ...  or  be  sure  to  check 
'None  of  the  above'  if  appropriate.)  [156  respondents] 


Category 

Count 

Monte  Carlo  simulation 

59 

Discrete  event  simulation  for  process  modeling 

38 

Markov  or  Petri-net  models 

6 

Probabilistic  modeling 

46 

Neural  networks 

4 

Optimization 

39 

Other  (Please  describe  briefly) 

5 

None  of  the  above 

36 

Don't  know 

7 

4.  Which  of  these  decision  techniques  are  used  in  your  organization?  (Please  select  as 
many  as  apply  ...  or  be  sure  to  check  'None  of  the  above'  if  appropriate.)  [156  respon¬ 
dents] 


Category 

Count 

Analytic  Hierarchy  Process  (AHP) 

15 

Real  options 

24 

Conjoint  analysis 

16 

Wide  band  Delphi 

65 

Weighted  multi  criteria  methods  (e.g.,  QFD  or 

Pugh) 

86 

Decision  trees 

86 

Other  (Please  describe  briefly) 

17 

None  of  the  above 

4 

Don't  know 

12 
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VI.  Challenges  &  Value  Added 


1.  Following  is  a  series  of  statements  about  the  kinds  of  technical  challenges  that  projects 
sometimes  face.  How  well  do  they  describe  your  organization?  (Please  select  one  for  each.) 


Initial  project  requirements  are  not  well  defined 


Category 

Count 

% 

1  =  Almost  entirely 

5 

3 

2  =  To  a  large  extent 

30 

21 

3  =  To  a  moderate  extent 

50 

35 

4  =  To  a  limited  extent 

43 

30 

5  =  Hardly  at  all 

14 

10 

6  =  Don't  know 

0 

0 

7  =  Not  applicable 

2 

1 

Total 

144 

Requirements  change  significantly  throughout  the  life  of  the  projects 


Category 

Count 

% 

1  =  Almost  entirely 

5 

3 

2  =  To  a  large  extent 

32 

22 

3  =  To  a  moderate  extent 

70 

49 

4  =  To  a  limited  extent 

34 

24 

5  =  Hardly  at  all 

3 

2 

6  =  Don't  know 

0 

0 

7  =  Not  applicable 

0 

0 

Total 

144 

There  is  little  or  no  precedent  for  the  kind  of  work  we  are  doing 


Category 

Count 

% 

1  =  Almost  entirely 

2 

1 

2  =  To  a  large  extent 

4 

3 

3  =  To  a  moderate  extent 

30 

21 

4  =  To  a  limited  extent 

65 

45 

5  =  Hardly  at  all 

38 

26 

6  =  Don't  know 

0 

0 

7  =  Not  applicable 

0 

0 

Total 

144 

Significant  constraints  are  placed  on  product  quality  attributes  (e.g.,  reliability,  scalability, 
security,  supportability,  etc.) 


Category 

Count 

% 

1  =  Almost  entirely 

10 

7 

2  =  To  a  large  extent 

38 

27 
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3  =  To  a  moderate  extent 

35 

24 

4  =  To  a  limited  extent 

42 

29 

5  =  Hardly  at  all 

14 

10 

6  =  Don't  know 

1 

1 

7  =  Not  applicable 

3 

2 

Total 

143 

The  size  of  the  development  effort  is  large 


Category 

Count 

% 

1  =  Almost  entirely 

8 

6 

2  =  To  a  large  extent 

40 

28 

3  =  To  a  moderate  extent 

58 

40 

4  =  To  a  limited  extent 

27 

19 

5  =  Hardly  at  all 

8 

6 

6  =  Don't  know 

2 

1 

7  =  Not  applicable 

1 

1 

Total 

144 

The  technology  needed  for  the  projects  is  not  mature 


Category 

Count 

% 

1  =  Almost  entirely 

0 

0 

2  =  To  a  large  extent 

13 

9 

3  =  To  a  moderate  extent 

27 

19 

4  =  To  a  limited  extent 

65 

45 

5  =  Hardly  at  all 

37 

26 

6  =  Don't  know 

0 

0 

7  =  Not  applicable 

2 

1 

Total 

144 

There  are  extensive  needs  for  interoperability  with  other  systems 


Category 

Count 

% 

1  =  Almost  entirely 

14 

10 

2  =  To  a  large  extent 

40 

28 

3  =  To  a  moderate  extent 

48 

33 

4  =  To  a  limited  extent 

33 

23 

5  =  Hardly  at  all 

6 

4 

6  =  Don't  know 

2 

1 

7  =  Not  applicable 

1 

1 

Total 

144 

Insufficient  resources  (e.g.,  people,  funding)  are  available  to  support  the  projects 


Category 

Count 

% 

1  =  Almost  entirely 

4 

3 

2  =  To  a  large  extent 

14 

10 

3  =  To  a  moderate  extent 

30 

21 

102  I  CMU/SEI-2008-TR-024 


4  =  To  a  limited  extent 

62 

43 

5  =  Hardly  at  all 

33 

23 

6  =  Don't  know 

0 

0 

7  =  Not  applicable 

1 

1 

Total 

144 

Insufficient  skiiis  and  subject  matter  expertise  are  avaiiabie  to  support  the  projects 


Category 

Count 

% 

1  =  Almost  entirely 

0 

0 

2  =  To  a  large  extent 

9 

6 

3  =  To  a  moderate  extent 

22 

15 

4  =  To  a  limited  extent 

67 

47 

5  =  Hardly  at  all 

45 

31 

6  =  Don't  know 

1 

1 

7  =  Not  applicable 

0 

0 

Total 

144 

Other  (Piease  describe  briefiy) 


Category 

Count 

% 

1  =  Almost  entirely 

1 

5 

2  =  To  a  large  extent 

0 

0 

3  =  To  a  moderate  extent 

1 

5 

4  =  To  a  limited  extent 

0 

0 

5  =  Hardly  at  all 

0 

0 

6  =  Don't  know 

2 

10 

7  =  Not  applicable 

17 

81 

Total 

21 

2.  Foiiowing  are  a  few  statements  about  the  possibie  effects  of  using  process  performance 
modeiing.  To  what  extent  do  they  describe  what  your  organization  has  experienced?  (Piease 
seiect  one  for  each.) 

Better  project  performance  (e.g.,  more  accurate  estimation,  reduced  cost,  shorter  cycie 
time  or  higher  productivity) 


Category 

Count 

% 

1  =  Almost  always 

33 

23 

2  =  Frequently 

69 

48 

3  =  About  half  the  time 

19 

13 

4  =  Occasionally 

19 

13 

5  =  Rarely  if  ever 

2 

1 

6  =  Worse,  not  better 

0 

0 

7  =  Don't  know 

1 

1 

8  =  Not  applicable 

0 

0 

Total 

143 
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Better  product  quality  (e.g.,  fewer  defects  or  improved  customer  satisfaction) 


Category 

Count 

% 

1  =  Almost  always 

34 

24 

2  =  Frequently 

69 

48 

3  =  About  half  the  time 

16 

11 

4  =  Occasionally 

19 

13 

5  =  Rarely  if  ever 

4 

3 

6  =  Worse,  not  better 

0 

0 

7  =  Don't  know 

2 

1 

8  =  Not  applicable 

0 

0 

Total 

144 

Fewer  project  failures 


Category 

Count 

% 

1  =  Almost  always 

32 

22 

2  =  Frequently 

61 

43 

3  =  About  half  the  time 

16 

11 

4  =  Occasionally 

20 

14 

5  =  Rarely  if  ever 

8 

6 

6  =  Worse,  not  better 

0 

0 

7  =  Don't  know 

2 

1 

8  =  Not  applicable 

4 

3 

Total 

143 

Better  tactical  decisions  about  the  adoption  or  improvement  of  work  processes  and  tech¬ 
nologies) 


Category 

Count 

% 

1  =  Almost  always 

16 

11 

2  =  Frequently 

65 

45 

3  =  About  half  the  time 

11 

8 

4  =  Occasionally 

32 

22 

5  =  Rarely  if  ever 

13 

9 

6  =  Worse,  not  better 

0 

0 

7  =  Don't  know 

3 

2 

8  =  Not  applicable 

3 

2 

Total 

143 
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Better  strategic  decision  making  (e.g.,  about  business  growth  or  profitabiiity) 


Category 

Count 

% 

1  =  Almost  always 

10 

7 

2  =  Frequently 

49 

34 

3  =  About  half  the  time 

17 

12 

4  =  Occasionally 

34 

24 

5  =  Rarely  if  ever 

18 

13 

6  =  Worse,  not  better 

0 

0 

7  =  Don't  know 

8 

6 

8  =  Not  applicable 

7 

5 

Total 

143 

Other  (Piease  describe  briefiy) 


Category 

Count 

% 

1  =  Almost  always 

2 

9 

2  =  Frequently 

1 

5 

3  =  About  half  the  time 

0 

0 

4  =  Occasionally 

0 

0 

5  =  Rarely  if  ever 

0 

0 

6  =  Worse,  not  better 

0 

0 

7  =  Don't  know 

5 

23 

8  =  Not  applicable 

14 

64 

Total 

22 

3.  How  often  are  process  performance  modei  predictions  used  to  inform  decision  making 
your  organization's  status  and  miiestone  reviews?  (Piease  seiect  one.) 


Category 

Count 

% 

1  Almost  always 

28 

20 

2  Frequently 

57 

40 

3  About  half  the  time 

27 

19 

4  Occasionally 

23 

16 

5  Rarely  if  ever 

7 

5 

6  Don't  know 

1 

1 

Total 

143 
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4.  Overall,  how  useful  have  process  performance  models  been  for  your  organization? 
(Please  select  one.) 


Category 

Count 

% 

1  =  Extremely  valuable  —  we  couldn't  do  our  work  properly 

without  them 

12 

8 

2  =  Very  valuable  —  we  have  obtained  much  useful  information 

from  them 

74 

51 

3  =  Mixed  value  —  we  have  obtained  useful  information  on  oc- 

casion 

55 

38 

4  =  Little  or  no  value 

3 

2 

5  =  It's  been  harmful,  not  helpful 

0 

0 

6  =  Don't  know 

0 

0 

Total 

144 

106  I  CMU/SEI-2008-TR-024 


VII.  Barriers  &  Facilitators  of  Effective  Measurement  &  Analysis 


1.  Which,  if  any,  of  the  foiiowing  have  been  major  obstacies  during  your  organization's 
journey  to  high  maturity?  (Piease  seiect  as  many  as  appiy  ...  or  be  sure  to  check  'None  of 
the  above'  if  appropriate.)  [156  respondents] 


Category 

Count 

We  focused  only  on  final  project  outcomes  rather  than  interim  out¬ 
comes 

29 

We  didn't  collect  data  frequently  enough  to  help  projects  make  mid¬ 
course  corrections 

36 

We  failed  to  collect  enough  contextual  information  for  proper  seg¬ 
mentation  and  stratification 

48 

We  failed  to  achieve  enough  consistency  in  our  measures  to  aggre¬ 
gate  and  disaggregate  them  properly  across  the  organization 

44 

We  failed  to  sufficiently  align  and  prioritize  our  measurement  and 
analysis  practices  with  our  business  and  technical  goals  and  objec¬ 
tives 

26 

We've  encountered  resistance  to  collecting  new  or  additional  data 
after  achieving  maturity  level  3 

42 

Our  management  thought  that  process  performance  modeling  would 
be  an  expensive  overhead  function  rather  than  an  essential  part  of 
project  work 

23 

We  spent  too  much  time  creating  reports  for  management  review 
instead  of  doing  thorough  analysis 

33 

We  emphasized  statistics  more  than  domain  knowledge  and  ended 
up  with  ineffective  models 

15 

We  didn't  provide  sufficient  mentoring  and  coaching  for  the  indi¬ 
viduals  responsible  for  developing  the  models 

35 

Our  process  performance  modelers  don't  have  sufficient  access  to 
people  with  statistical  expertise 

35 

Other  (Please  describe  briefly) 

14 

None  of  the  above 

31 

Don't  know 

1 
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2.  Following  is  a  series  of  statements  that  are  made  in  some  organizations  about  the  use  of 
process  performance  modeling.  How  well  do  they  describe  your  organization?  (Please  se¬ 
lect  one  for  each.) 


We  have  trouble  doing  process  performance  modeling  because  it  takes  too  long  to  accumu¬ 
late  enough  historical  data _ _ _ 


Category 

Count 

% 

1  =  Almost  entirely 

5 

3 

2  =  To  a  large  extent 

31 

21 

3  =  To  a  moderate  extent 

34 

23 

4  =  To  a  limited  extent 

36 

25 

5  =  Hardly  at  all 

30 

21 

6  =  Don't  know 

0 

0 

7  =  Not  applicable 

9 

6 

Total 

145 

Doing  process  performance  modeling  has  become  an  accepted  way  of  doing  business  here 


Category 

Count 

% 

1  =  Almost  entirely 

14 

10 

2  =  To  a  large  extent 

37 

26 

3  =  To  a  moderate  extent 

51 

35 

4  =  To  a  limited  extent 

32 

22 

5  =  Hardly  at  all 

8 

6 

6  =  Don't  know 

1 

1 

7  =  Not  applicable 

2 

1 

Total 

145 

We  make  our  decisions  about  the  models  we  build  without  sufficient  participation  by  man¬ 
agement  or  other  important  stakeholders _ _ _ 


Category 

Count 

% 

0  =  none  selected 

2 

1 

1  =  Almost  entirely 

12 

8 

2  =  To  a  large  extent 

18 

12 

3  =  To  a  moderate  extent 

51 

35 

4  =  To  a  limited  extent 

55 

38 

5  =  Hardly  at  all 

1 

1 

6  =  Don't  know 

6 

4 

7  =  Not  applicable 

145 

Total 

2 

1 
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We  have  trouble  convincing  management  about  value  of  doing  process  performance 
modeling 


Category 

Count 

% 

1  =  Almost  entirely 

2 

1 

2  =  To  a  large  extent 

7 

5 

3  =  To  a  moderate  extent 

22 

15 

4  =  To  a  limited  extent 

39 

27 

5  =  Hardly  at  all 

70 

48 

6  =  Don't  know 

4 

3 

7  =  Not  applicable 

5 

3 

Total 

145 

The  messenger  has  been  shot  for  delivering  bad  news  based  on  process  performance 
model  predictions 


Category 

Count 

% 

1  =  Almost  entirely 

4 

3 

2  =  To  a  large  extent 

4 

3 

3  =  To  a  moderate  extent 

12 

8 

4  =  To  a  limited  extent 

17 

12 

5  =  Hardly  at  all 

84 

58 

6  =  Don't  know 

1 

1 

7  =  Not  applicable 

27 

19 

Total 

145 

We  thought  we  knew  what  was  driving  process  performance,  but  process  performance 
modeling  has  taught  us  otherwise 


Category 

Count 

% 

0  =  none  selected 

1 

1 

1  =  Almost  entirely 

13 

9 

2  =  To  a  large  extent 

37 

26 

3  =  To  a  moderate  extent 

48 

33 

4  =  To  a  limited  extent 

24 

17 

5  =  Hardly  at  all 

10 

7 

6  =  Don't  know 

11 

8 

7  =  Not  applicable 

144 

Total 

1 

1 

Our  managers  want  to  know  when  things  are  off-track 


Category 

Count 

% 

1  =  Almost  entirely 

52 

36 

2  =  To  a  large  extent 

50 

35 

3  =  To  a  moderate  extent 

24 

17 

4  =  To  a  limited  extent 

9 

6 

5  =  Hardly  at  all 

3 

2 

6  =  Don't  know 

2 

1 

7  =  Not  applicable 

4 

3 

Total 

144 
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Our  managers  are  less  willing  to  fund  new  work  when  the  outcome  is  uncertain 


Category 

Count 

% 

1  =  Almost  entirely 

4 

3 

2  =  To  a  large  extent 

33 

23 

3  =  To  a  moderate  extent 

33 

23 

4  =  To  a  limited  extent 

33 

23 

5  =  Hardly  at  all 

28 

19 

6  =  Don't  know 

4 

3 

7  =  Not  applicable 

9 

6 

Total 

144 

We  use  data  mining  when  similar  but  not  identical  electronic  records  exist 


Category 

Count 

% 

1  =  Almost  entirely 

4 

3 

2  =  To  a  large  extent 

10 

7 

3  =  To  a  moderate  extent 

31 

21 

4  =  To  a  limited  extent 

40 

28 

5  =  Hardly  at  all 

25 

17 

6  =  Don't  know 

18 

12 

7  =  Not  applicable 

17 

12 

Total 

145 

We  do  real  time  sampling  of  current  processes  when  historical  data  are  not  available 


Category 

Count 

% 

1  =  Almost  entirely 

5 

3 

2  =  To  a  large  extent 

23 

16 

3  =  To  a  moderate  extent 

36 

25 

4  =  To  a  limited  extent 

43 

30 

5  =  Hardly  at  all 

21 

14 

6  =  Don't  know 

3 

2 

7  =  Not  applicable 

14 

10 

Total 

145 

We  create  our  baselines  from  paper  records  for  previously  unmeasured  attributes 


Category 

Count 

% 

1  =  Almost  entirely 

2 

1 

2  =  To  a  large  extent 

1 

1 

3  =  To  a  moderate  extent 

18 

13 

4  =  To  a  limited  extent 

34 

24 

5  =  Hardly  at  all 

55 

38 

6  =  Don't  know 

10 

7 

7  =  Not  applicable 

24 

17 

Total 

144 

3.  What  have  been  the  greatest  barriers  faced  by  your  organization  during  its  journey  to 
high  maturity?  What  have  you  done  to  overcome  them?  (Please  describe  fully) 
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Facsimile  of  Survey  Announcement 

Measurement  and  Analysis  in  CMMI  High  Maturity  Organizations 
To  the  attention  of:  <Invitee> 

From:  Mike  Phillips,  CMMI  Program  Manager,  Software  Engineering  Institute 
Date:  May  6,  2008 

There  has  been  a  great  deal  of  discussion  recently  about  high  maturity  processes  and  project  per¬ 
formance.  Clarification  is  needed  along  with  good  examples  of  what  has  worked  well  and  what 
has  not.  Measurement  and  analysis  activities  are  key  ingredients. 

As  part  of  the  SETs  ongoing  research  and  development  effort,  we  are  conducting  a  survey  on  the 
use  of  measurement  and  analysis  in  high  maturity  organizations.  Because  of  your  organization’s 
leadership  in  this  area,  you  will  receive  an  invitation  soon  [dl]  inviting  participation  in  the  survey. 
As  always,  any  information  that  could  identify  you  or  your  organization  will  be  held  in  strict  con¬ 
fidence  by  the  SEI  for  research  purposes  only. 

Participants  in  the  survey  will  receive  an  early  copy  of  the  results.  We  hope  that  the  analysis  will 
allow  you  to  make  useful  comparisons  with  similar  organizations  and  provide  practical  guidance 
for  continuing  improvement  in  your  own  organization.  (Eor  a  similar  example  you  may  visit 
http://www.sei.cmu.edu/publications/documents/07.reports/07sr014.html.) 

Many  of  the  questions  center  on  process  performance  modeling.  You  may  wish  to  assign  comple¬ 
tion  of  the  questionnaire  to  someone  who  is  more  familiar  with  the  day-to-day  details  of  the  or¬ 
ganization’s  measurement  related  activities.  (Pertinent  CMMI  excerpts  are  at 
https://seir.sei.cmu.edu/feedback/background.htm.) 

I  would  appreciate  knowing  if  you  are  unable  or  unwilling  to  complete  the  qnestionnaire  at  this 
time.  We  need  to  hear  from  everyone,  and  insights  on  disappointments  may  be  as  valuable  for 
improving  our  performance  as  those  from  organizations  who  have  found  our  current  approaches 
useful.  Simply  reply  to  this  message;  I  will  receive  it  in  my  personal  email  acconnt. 

Thank  you  in  advance. 
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Facsimile  of  Survey  Invitation 

Measurement  and  Analysis  in  CMMI  High  Maturity  Organizations 
To  the  attention  of:  <Invitee> 

Date:  May  6,  2008 

As  you  hopefully  have  seen  earlier  today  in  an  email  from  Mike  Phillips,  the  CMMI  Program 
Manager,  the  SEI  is  conducting  a  survey  on  the  use  of  measurement  and  analysis  in  high  maturity 
organizations.  A  relatively  limited  number  of  organizations  have  experience  in  this  area,  so  we 
would  greatly  appreciate  having  yours  participate.  We  need  to  hear  from  everyone  for  the  results 
to  be  accurate  and  useful. 

Please  answer  from  the  perspective  of  your  organizational  unit  that  most  recently  had  a  CMMI- 
based  high  maturity  appraisal.  As  Mike  mentioned,  much  of  the  questionnaire  centers  on  process 
performance  modeling.  Please  feel  free  to  consult  with  or  delegate  completion  of  the  question¬ 
naire  to  someone  who  may  be  more  familiar  with  the  day-to-day  details  of  the  organization's  mea¬ 
surement  related  activities. 

You  will  find  your  personalized  form  on  the  World  Wide  Web  at 
https://seir.sei. emu. edu/feedback/HighMaturity2008.asp?ID=<random>. 

Please  be  sure  to  complete  it  at  your  earliest  convenience — right  now  if  you  can  make  the  time. 
You  or  your  designee  may  return  to  that  URL  and  continue  completing  the  questionnaire  at  any 
time.  You  also  may  save  your  work  at  any  time.  Answering  all  of  the  questions  should  take  about 
20  to  30  minutes. 

Please  complete  the  questionnaire  as  candidly  and  completely  as  you  possibly  can.  As  always,  any 
information  that  could  identify  you  or  your  organization  will  be  held  in  strict  confidence  by  the 
SEI  under  promise  of  non  disclosure.  The  results  will  be  reported  in  sununary  aggregate  form 
without  attribution.  Individual  replies  will  be  seen  only  by  selected  members  of  the  analysis  team, 
for  data  management  purposes  only.  You  may  rest  assured  that  your  individual  replies  will  not  be 
used  or  known  for  appraisal  audit  purposes. 

Please  feel  free  to  contact  us  at  sei-analysis@sei.cmu.edu  if  you  have  any  questions  or  concerns 
about  this  work,  or  have  any  trouble  completing  your  form  over  the  web.  We'll  get  back  to  you 
right  away. 

Thank  you  in  advance. 

Bob  Stoddard  and  Dennis  Goldenson 
for  the  Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
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Facsimile  of  First  Reminder 

Reminder  -  Measurement  and  Analysis  in  High  Maturity  Organizations 
To  the  attention  of:  <Invitee> 

Date:  May  21,  2008 

About  two  weeks  ago,  we  asked  you  to  complete  a  questionnaire  on  the  use  of  measurement  and 
analysis  in  high  maturity  organizations.  As  you  know,  yours  is  one  of  a  still  small  number  of  or¬ 
ganizations  with  experience  in  this  area,  so  your  participation  is  extremely  important.  We  need  to 
hear  from  everyone  to  ensure  that  the  survey  results  are  accurate  and  useful. 

Please  do  complete  and  submit  your  questionnaire  by  June  4.  As  a  reminder,  you  may  find  it  at 
https://seir.sei. emu. edu/feedback/HighMaturity2008.asp?ID=<random>. 

Recall  that  much  of  the  questionnaire  centers  on  process  performance  modeling.  So  please  feel 
free  to  consult  with  or  delegate  completion  of  the  questionnaire  to  someone  who  may  be  more 
familiar  with  the  day-to-day  details  of  the  organization’s  measurement  related  activities. 

Answering  all  of  the  questions  should  take  about  20  to  30  minutes.  As  always,  your  candid  an¬ 
swers  and  anything  that  could  identify  you  or  your  organization  will  be  held  in  strict  confidence 
by  the  SET  An  early  summary  of  the  results  will  be  sent  to  those  who  have  completed  their  ques¬ 
tionnaires.  We  hope  that  it  will  prove  to  be  useful  for  you. 

Please  feel  free  to  contact  us  at  sei-analysis@sei.cmu.edu  if  you  have  any  questions  or  concerns 
about  this  work,  or  have  any  trouble  completing  your  form  over  the  web.  We’ll  do  our  best  to  re¬ 
solve  any  questions  or  concerns  that  you  may  have. 

Once  again,  thank  you  for  your  contribution  and  cooperation  in  this  effort. 

Bob  Stoddard  and  Dennis  Goldenson 
for  the  Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
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Facsimile  of  Second  Reminder 

Second  Reminder  -  Measurement  and  Analysis  in  High  Maturity  Organizations 
To  the  attention  of:  <Invitee> 

Date:  June  6,  2008 

As  you  know,  the  Software  Engineering  Institute  has  been  conducting  a  survey  on  the  use  of  mea¬ 
surement  and  analysis  in  high  maturity  organizations.  Naturally  we  hope  that  the  results  will  help 
you  better  judge  your  own  progress  relative  to  the  successes  and  challenges  reported  by  others. 

We  have  been  encouraged  by  the  number  of  responses  we  have  received  so  far;  however  we  have 
not  yet  received  a  reply  from  you. 

Please  do  make  about  20  to  30  minutes  time  to  complete  your  questionnaire  as  soon  as  you  possi¬ 
bly  can,  by  June  18  if  at  all  possible.  Remember  that  you  may  return  to  complete  it  over  more  than 
one  session  if  that's  easier  for  you. 

Once  again,  your  personalized  form  is  on  the  World  Wide  Web  at 
https://seir.sei. emu. edu/feedback/HighMaturity2008.asp?ID=<random>. 

Thank  you  for  your  contribution  and  cooperation  with  this  effort.  We  hope  that  the  results  will 
prove  to  be  valuable  to  you. 

Most  sincerely. 

Bob  Stoddard  and  Dennis  Goldenson 
for  the  Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
sei-analysis  @  sei.cmu.edu 
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Facsimile  of  Final  Reminder 

Final  Reminder  -  Measurement  and  Analysis  in  High  Maturity  Organizations 
To  the  attention  of:  <Invitee> 

Date:  June  19,  2008 

We  are  writing  to  you  about  the  Software  Engineering  Institute's  survey  on  the  use  and  perform¬ 
ance  outcomes  of  measurement  and  analysis  in  high  maturity  organizations.  We  have  not  yet  re¬ 
ceived  your  completed  questionnaire. 

The  survey  is  the  most  comprehensive  of  it's  kind  yet  done,  and  we  expect  it  to  he  of  considerable 
value  to  organizations  that  wish  to  continue  improving  their  measurement  practices.  Of  course, 
your  answers  are  needed  to  make  the  survey  an  accurate  representation  of  the  state  of  the  practice 
in  high  maturity  organizations. 

Please  do  make  time  to  complete  your  questionnaire  as  soon  as  you  possibly  can,  by  the  end  of 
this  month  at  latest.  We  will  let  contributors  know  as  soon  as  the  results  are  ready. 

Once  again,  your  personalized  form  is  on  the  World  Wide  Web  at 
https://seir.sei. emu. edu/feedback/HighMaturity2008.asp?ID=<random>. 

Remember  that  you  may  return  to  complete  it  over  more  than  one  session  if  that's  easier  for  you. 
Of  course,  your  answers  will  be  held  in  strict  confidence. 

Thank  you  again  in  advance  for  your  contribution  and  cooperation  with  this  effort.  We  hope  that 
the  results  will  prove  to  be  valuable  to  you. 

Most  sincerely, 

Bob  Stoddard  and  Dennis  Goldenson 
for  the  Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
sei-analysis  @  sei.cmu.edu 
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Appendix  B:  Questionnaire  for  the  Generai  Popuiation 
Survey 


This  appendix  contains  a  listing  of  all  of  the  survey  questions  that  have  forced-choice,  closed- 
ended  answers.  It  is  annotated  with  the  number  of  responses  for  each  answer  and  the  percentage 
of  the  total  answers  that  each  answer  represents.  Facsimiles  of  the  invitations  and  reminder  letters 
sent  to  the  survey  respondents  follow  the  question  and  answer  listing  in  this  Appendix.^® 


^  We  also  sent  very  similar  personalized  reminders  to  those  who  had  begun  but  not  yet  completed  their  question¬ 
naires.  They  are  not  reproduced  here  in  the  interests  of  space. 
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^gftwanp  Engfn^rrtq 
Carnegie  Me  Hon 

The  State  of  Measurement  &  Analysis  Practice:  2008  Survey 


1.  Do  you  work  for  or  support  a  software  or  systems  engineering  organization  as  we  have 
defined  it  here?  (Piease  seiect  one.) 


Count 

o/o 

Yes  -  Currently 

267 

79 

No  -  But  I  have  within  the  past  two  years 

16 

5 

No  -  I  do  not 

52 

15 

Total 

283 

2.  Which  of  the  foiiowing  best  describes  the  roie  you  piay  in  your  organization?  (Piease  se¬ 
iect  one.) 


Category 

Count 

% 

Executive  or  senior  manager 

43 

15 

Middle  manager  (e.g.,  program  or  product  line) 

33 

12 

Project  manager 

41 

15 

Project  engineer  or  other  technical  staff 

30 

11 

Process  or  quality  engineer 

90 

32 

Measurement  specialist 

8 

3 

Other  (Please  describe  briefly) 

36 

13 

Total 

281 

3.  How  is  your  organization  best  described?  (Piease  seiect  one.) 


Category 

Count 

% 

Commercial  off  the  shelf 

21 

7 

Contracted  new  development 

62 

22 

In-house  or  proprietary  development  or  maintenance 

78 

28 

Defense  contractor 

46 

16 

Other  government  contractor 

12 

4 

Department  of  Defense  or  military  organization 

9 

3 

Other  government  agency 

11 

4 

Other  (Please  describe  briefly) 

42 

15 

Total 

281 
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4.  In  what  country  is  your  organization  primariiy  iocated?  (Piease  seiect  one.) 


Count 

% 

United  States 

131 

46 

Canada 

2 

1 

China 

7 

2 

France 

6 

2 

Germany 

5 

2 

India 

32 

11 

Japan 

9 

3 

Netherlands 

2 

1 

United  Kingdom 

8 

3 

All  Others 

80 

28 

Total 

282 

5.  Approximate^  how  many  fuii-time  empioyees  work  in  your  organization?  (Piease 
seiect  one.) 


Category 

Count 

% 

25  or  fewer 

40 

14 

26-50 

19 

7 

51-75 

12 

4 

76-100 

16 

6 

101-200 

26 

9 

201-300 

19 

7 

301-500 

24 

9 

501-1000 

29 

10 

1001-2000 

17 

6 

More  than  2000 

75 

27 

Total 

277 

6.  To  the  best  of  your  knowiedge,  what  is  the  maturity  ievei  of  your  organization?  (Piease 
seiect  one.) _ _ _ 


Category 

Count 

% 

CMMI  Maturity  Level  1  (Initial) 

47 

17 

Close  To  Maturity  Level  2 

43 

15 

CMMI  Maturity  Level  2  (Managed) 

17 

6 

Close  To  Maturity  Level  3 

34 

12 

CMMI  Maturity  Level  3  (Defined) 

61 

21 

CMMI  Maturity  Level  4  (Quantitatively  Managed) 

9 

3 

CMMI  Maturity  Level  5  (Optimizing) 

46 

16 

Don't  know 

27 

10 

Total 

284 
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7.  How  would  you  best  describe  your  involvement  with  measurement?  (Please  select  one.) 


Category 

Count 

% 

I  am  a  provider  of  measurement-based  information 

43 

15 

I  am  a  user  (consumer)  of  measurement-based  information 

42 

15 

*  I  am  both  a  provider  and  user  (consumer)  of  measurement-based  in¬ 
formation 

157 

56 

*  Other  (Please  describe  briefly) 

40 

14 

Total 

282 

*  -  proceed  to  Section  VI 

8.  How  frequently  is  measurement  and  analysis  used  in  your  organization?  (Please  select 
one.) 


Category 

Count 

% 

Routinely 

176 

62 

Occasionally 

56 

20 

*  Rarely  if  ever 

37 

13 

*  Don't  know 

15 

5 

Total 

284 

*  -  proceed  to  Section  VI 
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II.  Resources  and  Infrastructure 

1.  Which  of  the  following  best  describes  how  measurement  related  work  is  staffed  in  your 
organization?  (Please  select  one.) 


Category 

Count 

% 

An  organization-wide,  division  or  similar  corporate  support  group  (e.g., 
an  engineering  process,  quality  assurance  or  measurement  group) 

90 

38 

Separate  groups  or  individuals  in  different  projects  or  other  organizational 
units  (e.g.,  project,  product  team  or  similar  work  groups) 

68 

29 

A  few  key  people  (or  one  person)  in  the  organization  who  are  measure¬ 
ment  experts 

54 

23 

Other  (Please  describe  briefly) 

24 

10 

Total 

236 

2.  How  often  are  qualified,  well-prepared  people  available  to  work  on  measurement  and 
analysis  in  your  organization  when  you  need  them,  i.e.,  people  with  sufficient  measure¬ 
ment  related  knowledge,  competence  or  statistical  sophistication?  (Please  select  one.) 


Category 

Count 

% 

Almost  always  (Greater  than  or  equal  to  80%) 

53 

23 

Frequently  (Greater  than  or  equal  to  60%) 

50 

21 

About  half  of  the  time  (Greater  than  40%  but  less  than  60%) 

36 

15 

Occasionally  (Less  than  or  equal  to  40%) 

50 

21 

Rarely  if  ever  (Less  than  or  equal  to  20%) 

44 

19 

Total 

233 
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3.  To  what  extent,  if  any,  has  the  lack  of  automated  support  made  measurement  related 
activities  difficult  for  your  organization?  (Please  select  one.) 


Category 

Count 

% 

Extensive  difficulty 

17 

7 

Substantial 

71 

30 

Moderate 

73 

31 

Limited 

37 

16 

Little  if  Any  difficulty 

22 

9 

Don't  know 

14 

6 

Total 

220 

4.  How  would  you  best  characterize  the  measurement  related  training  that  is  available  in 
your  organization?  (Please  select  one.) 


Category 

Count 

% 

Excellent 

14 

6 

Good 

53 

23 

Adequate 

50 

21 

Fair 

60 

26 

Poor 

57 

24 

Total 

234 
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III.  Value  Added 


1.  In  general,  how  valuable  has  measurement  and  analysis  been  to  your  organization? 
(Please  select  one.) 


Category 

Count 

% 

Extremely  valuable  —  we  couldn't  do  our  work  properly  without  it 

30 

13 

Very  valuable  —  we  have  obtained  much  useful  information  from  it 

80 

35 

Mixed  value  —  we  have  obtained  useful  information  on  occasion 

96 

42 

Little  or  no  value 

13 

5 

It's  been  harmful,  not  helpful 

1 

0 

Don't  know 

11 

5 

Total 

231 

2.  Following  are  a  few  statements  about  the  possible  effects  of  measurement  and  analysis. 
To  what  extent  do  they  describe  what  your  organization  has  experienced?  (Please  select 
one  for  each.) 


Better  project  performance  (e.g.,  more  accurate  estimation,  reduced  cost,  shorter  cycle 
time,  higher  productivity) 


Category 

Count 

% 

Almost  always 

44 

19 

Frequently 

65 

29 

About  half  the  time 

33 

14 

Occasionally 

53 

23 

Rarely  if  ever 

18 

8 

"Worse,  not  better" 

2 

1 

Don't  know 

5 

2 

Not  applicable 

8 

4 

Total 

228 

Better  product  quality  (e.g.,  fewer  defects.  Improved  customer  satisfaction) 


Category 

Count 

% 

Almost  always 

54 

24 

Frequently 

72 

32 

About  half  the  time 

34 

15 

Occasionally 

30 

13 

Rarely  if  ever 

18 

8 

"Worse,  not  better" 

1 

0 

Don't  know 

8 

4 

Not  applicable 

8 

4 

Total 

225 
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Better  tactical  decisions  about  the  adoption  or  improvement  of  work  processes  and 
technologies 


Category 

Count 

% 

Almost  always 

31 

14 

Frequently 

62 

28 

About  half  the  time 

25 

11 

Occasionally 

58 

26 

Rarely  if  ever 

31 

14 

"Worse,  not  better" 

1 

0 

Don't  know 

9 

4 

Not  applicable 

7 

3 

Total 

224 

Better  strategic  decision  making  (e.g.,  about  business  growth  and  profitability) 


Category 

Count 

% 

Almost  always 

24 

11 

Frequently 

48 

22 

About  half  the  time 

29 

13 

Occasionally 

60 

27 

Rarely  if  ever 

32 

14 

"Worse,  not  better" 

2 

1 

Don't  know 

14 

6 

Not  applicable 

13 

6 

Total 

222 

Other  (Please  describe  briefly) 


Category 

Count 

% 

Almost  always 

6 

18 

Frequently 

6 

18 

About  half  the  time 

1 

3 

Occasionally 

2 

6 

Rarely  if  ever 

1 

3 

"Worse,  not  better" 

6 

18 

Don't  know 

11 

33 

Not  applicable 

6 

18 

Total 

33 

3.  In  what  specific  ways  has  the  use  of  measurement  and  analysis  been  most  helpful, 
harmful,  to  your  organization?  (Please  describe  briefly) 


123  I  CMU/SEI-2008-TR-024 


IV.  Alignment  and  Coordination  of  Measurement  Activities 

1.  Do  you  agree  or  disagree  with  the  two  foiiowing  statements?  (Piease  seiect  one 
for  each.) 

I  generaiiy  find  the  definitions  of  the  measures  that  are  used  in  my  organization  to  be 
understandabie  and  consistent. 


Category 

Count 

% 

Strongly  agree 

40 

18 

Agree 

91 

40 

Somewhat  agree 

51 

22 

Not  sure 

5 

2 

Somewhat  disagree 

18 

8 

Disagree 

12 

5 

Strongly  disagree 

4 

2 

Not  applicable 

6 

3 

Total 

227 

I  usuaiiy  can  understand  and  interpret  the  measurement  resuits  that  I  see. 


Category 

Count 

% 

Strongly  agree 

42 

19 

Agree 

113 

50 

Somewhat  agree 

38 

17 

Not  sure 

6 

3 

Somewhat  disagree 

10 

4 

Disagree 

5 

2 

Strongly  disagree 

3 

1 

Not  applicable 

7 

3 

Total 

224 

2.  To  what  extent,  if  any,  do  concerns  about  data  accuracy  and  quaiity  make  measurement 
and  anaiysis  difficuit  for  your  organization?  (Piease  seiect  one.) 


Category 

Count 

% 

Extensive  difficulty 

7 

3 

Substantial 

71 

31 

Moderate 

89 

39 

Limited 

32 

14 

Little  if  Any  difficulty 

21 

9 

Don't  know 

8 

4 

Total 

228 
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3.  How  well  do  the  following  statements  describe  the  team  with  whom  you  work  most 
closely?  (Please  select  one  for  each.) 

There  exist  measurable  criteria  for  the  products  and  services  to  which  I  contribute. 


Category 

Count 

% 

Almost  always 

62 

27 

Frequently 

79 

35 

About  half  the  time 

23 

10 

Occasionally 

39 

17 

Rarely  if  ever 

19 

8 

Don't  know 

4 

2 

Total 

226 

I  use  measurement  to  understand  the  quality  of  the  products  and/or  services  that  I  work 
on. 


Category 

Count 

% 

Almost  always 

76 

33 

Frequently 

73 

32 

About  half  the  time 

26 

11 

Occasionally 

28 

12 

Rarely  if  ever 

23 

10 

Don't  know 

3 

1 

Total 

229 

My  team  follows  a  documented  process  for  collecting  measurement  data. 


Category 

Count 

% 

Almost  always 

97 

43 

Frequently 

50 

22 

About  half  the  time 

28 

12 

Occasionally 

24 

11 

Rarely  if  ever 

25 

11 

Don't  know 

4 

2 

Total 

228 

My  team  follows  a  documented  process  for  reporting  measurement  data  to  management. 


Category 

Count 

% 

Almost  always 

104 

46 

Frequently 

45 

20 

About  half  the  time 

24 

11 

Occasionally 

24 

11 

Rarely  if  ever 

26 

11 

Don't  know 

4 

2 

Total 

227 
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Corrective  action  is  taken  when  measurement  data  indicate  that  a  threshoid  has  been  ex¬ 
ceeded.  (By  "threshoid"  we  mean  a  target  or  boundary  that  when  exceeded  is  evidence 
that  a  risk  or  probiem  exists.) 


Category 

Count 

% 

Almost  always 

73 

32 

Frequently 

69 

30 

About  half  the  time 

26 

11 

Occasionally 

30 

13 

Rarely  if  ever 

26 

11 

Don't  know 

4 

2 

Total 

228 

I  understand  the  purposes  for  the  data  I  coiiect  or  report. 


Category 

Count 

% 

Almost  always 

150 

66 

Frequently 

46 

20 

About  half  the  time 

16 

7 

Occasionally 

10 

4 

Rarely  if  ever 

4 

2 

Don't  know 

3 

1 

Total 

229 

V.  Measures  Used 

1.  Approximateiy  how  often  are  the  foiiowing  kinds  of  project  and  organizationai  measure¬ 
ment  resuits  reported  in  your  organization?  (Piease  seiect  one  for  each.) 

Staff  adherence  to  deveiopment  work  processes 


Category 

Count 

% 

Regularly 

99 

44 

Frequently 

34 

15 

Occasionally 

41 

18 

Rarely  if  ever 

41 

18 

Don't  know 

5 

2 

Not  applicable 

4 

2 

Total 

224 
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Cost  performance  or  other  measures  of  budget  predictability 


Category 

Count 

% 

Regularly 

114 

51 

Frequently 

41 

18 

Occasionally 

33 

15 

Rarely  if  ever 

25 

11 

Don't  know 

8 

4 

Not  applicable 

3 

1 

Total 

224 

Schedule  performance,  milestone  satisfaction  or  other  measures  of  schedule  predictability 


Category 

Count 

% 

Regularly 

124 

55 

Frequently 

53 

24 

Occasionally 

28 

13 

Rarely  if  ever 

13 

6 

Don't  know 

3 

1 

Not  applicable 

3 

1 

Total 

224 

Accuracy  of  estimates,  e.g.,  effort,  cost  or  schedule 


Category 

Count 

% 

Regularly 

59 

26 

Frequently 

58 

26 

Occasionally 

57 

25 

Rarely  if  ever 

40 

18 

Don't  know 

7 

3 

Not  applicable 

3 

1 

Total 

224 

Product  cycle  time,  time  to  market,  or  delivery  rate 


Category 

Count 

% 

Regularly 

56 

25 

Frequently 

48 

21 

Occasionally 

45 

20 

Rarely  if  ever 

44 

21 

Don't  know 

19 

8 

Not  applicable 

12 

5 

Total 

224 
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Business  growth  and  profitability  (e.g.,  market  share,  revenue  generated,  profits,  or  return 
on  investment) 


Category 

Count 

% 

Regularly 

73 

33 

Frequently 

34 

15 

Occasionally 

34 

15 

Rarely  if  ever 

39 

17 

Don't  know 

29 

13 

Not  applicable 

15 

7 

Total 

224 

2.  Approximately  how  often  are  the  following  kinds  of  product  and  quality  measurement 
results  reported  in  your  organization?  (Please  select  one  for  each.) 

Product  requirements  or  architectures  (e.g.,  completion  of  customer  and  technical  re¬ 
quirements,  or  features  delivered  as  planned) 


Category 

Count 

% 

Regularly 

84 

38 

Frequently 

53 

24 

Occasionally 

45 

20 

Rarely  if  ever 

30 

13 

Don't  know 

8 

4 

Not  applicable 

3 

1 

Total 

223 

Effort  applied  to  tasks  (e.g.,  productivity,  rework,  and  cost  of  quality  or  poor  quality) 


Category 

Count 

% 

Regularly 

89 

40 

Frequently 

50 

22 

Occasionally 

39 

17 

Rarely  if  ever 

38 

17 

Don't  know 

5 

2 

Not  applicable 

3 

1 

Total 

224 

Defect  density  (e.g.,  numbers  of  defects  identified  pre  and  post  release) 


Category 

Count 

% 

Regularly 

89 

40 

Frequently 

37 

17 

Occasionally 

43 

19 

Rarely  if  ever 

43 

19 

Don't  know 

6 

3 

Not  applicable 

5 

3 

Total 

223 
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Defect  phase  containment  (i.e.,  early  detection  and  removal) 


Category 

Count 

% 

Regularly 

53 

24 

Frequently 

42 

19 

Occasionally 

54 

24 

Rarely  if  ever 

59 

26 

Don't  know 

8 

4 

Not  applicable 

8 

4 

Total 

224 

Quality  attributes  (e.g.,  maintainability,  interoperability,  portability,  usability,  reliability, 
complexity,  criticality,  reusability,  or  durability) 


Category 

Count 

% 

Regularly 

33 

15 

Frequently 

26 

12 

Occasionally 

66 

29 

Rarely  if  ever 

71 

32 

Don't  know 

15 

7 

Not  applicable 

13 

6 

Total 

222 

Customer  satisfaction  (e.g.,  satisfaction  with  staff  responsiveness  or  fitness  for  use  of  the 
delivered  product). 


Category 

Count 

% 

Regularly 

71 

32 

Frequently 

51 

23 

Occasionally 

58 

26 

Rarely  if  ever 

30 

13 

Don't  know 

8 

4 

Not  applicable 

5 

2 

Total 

223 
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VI.  Use  of  Measurement 


Following  is  a  series  of  statements  about  the  use  of  measurement  and  analysis  in  organiza¬ 
tions.  How  well  do  they  describe  your  organization?  (Please  select  one  for  each.) 


The  effort  required  for  people  to  submit  data  is  often  considered  to  be  onerous  or 
burdensome. 


Category 

Count 

% 

Almost  entirely 

23 

9 

To  a  large  extent 

70 

28 

To  a  moderate  extent 

68 

27 

To  a  limited  extent 

43 

17 

Hardly  at  all 

29 

12 

Don't  know 

13 

5 

Not  applicable 

5 

2 

Total 

251 

Measurement  data  and  analysis  results  are  generally  understandable  and  easily 
in  ter  pretable. 


Category 

Count 

% 

Almost  entirely 

33 

13 

To  a  large  extent 

81 

33 

To  a  moderate  extent 

58 

23 

To  a  limited  extent 

46 

18 

Hardly  at  all 

14 

6 

Don't  know 

14 

6 

Not  applicable 

3 

1 

Total 

249 

The  way  measurement  data  are  collected  and  used  is  often  considered  to  be  inappropriate 
by  the  people  who  must  provide  the  necessary  information  (e.g.,  irrelevant  to  their  own 
work,  or  used  to  unfairly  evaluate  performance  by  individual  people  or  projects). 


Category 

Count 

% 

Almost  entirely 

14 

6 

To  a  large  extent 

44 

18 

To  a  moderate  extent 

49 

20 

To  a  limited  extent 

59 

24 

Hardly  at  all 

54 

22 

Don't  know 

22 

9 

Not  applicable 

7 

3 

Total 

249 

The  measurement  data  that  we  collect  are  in  fact  analyzed  on  a  regular  basis. 
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Category 

Count 

% 

Almost  entirely 

51 

20 

To  a  large  extent 

80 

32 

To  a  moderate  extent 

42 

17 

To  a  limited  extent 

36 

14 

Hardly  at  all 

24 

10 

Don't  know 

11 

4 

Not  applicable 

7 

3 

Total 

251 

Measurement  and  data  analysis  are  an  integral  part  of  the  way  we  normally  do  business. 


Category 

Count 

% 

Almost  entirely 

40 

16 

To  a  large  extent 

68 

27 

To  a  moderate  extent 

48 

19 

To  a  limited  extent 

42 

17 

Hardly  at  all 

34 

14 

Don't  know 

11 

4 

Not  applicable 

5 

2 

Total 

248 

The  measurements  that  we  collect  aren't  very  relevant  to  the  business  and  development 
decisions  that  we  face. 


Category 

Count 

% 

Almost  entirely 

9 

4 

To  a  large  extent 

34 

14 

To  a  moderate  extent 

41 

16 

To  a  limited  extent 

55 

22 

Hardly  at  all 

82 

33 

Don't  know 

14 

6 

Not  applicable 

14 

6 

Total 

249 

The  need  for  objective  evidence  about  quality  and  performance  is  highly  valued  in  our 
organization. 


Category 

Count 

% 

Almost  entirely 

57 

23 

To  a  large  extent 

78 

31 

To  a  moderate  extent 

45 

18 

To  a  limited  extent 

31 

12 

Hardly  at  all 

26 

10 

Don't  know 

9 

4 

Not  applicable 

3 

1 

Total 

249 
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There  is  resistance  to  doing  measurement  around  here  (e.g.,  peopie  think  of  it  as 
unnecessary,  extra  work,  unfair,  or  an  imposition  on  the  way  they  do  their  work). 


Category 

Count 

% 

Almost  entirely 

23 

9 

To  a  large  extent 

50 

20 

To  a  moderate  extent 

56 

22 

To  a  limited  extent 

58 

23 

Hardly  at  all 

48 

19 

Don't  know 

11 

4 

Not  applicable 

4 

2 

Total 

250 
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Facsimile  of  Survey  Invitation 

The  State  of  Software  Measurement  Practice  -  2008 
To  the  attention  of:  <Invitee> 

From:  Measurement  and  Analysis  Team 
Date:  May  7,  2008 

The  Software  Engineering  Institute  (SEI)  is  conducting  its  third  annual  survey  about  the  state  of 
measurement  and  analysis  practice.  The  results  will  be  used  to  provide  guidance  for  future  meas¬ 
urement  efforts,  and  they  will  allow  valuable  comparisons  to  be  made  among  organizations  simi¬ 
lar  to  your  own. 

You  are  part  of  a  carefully  chosen  sample  for  the  survey.  Your  participation  is  necessary  for  the 
results  to  be  accurate  and  useful,  even  if  your  organization  rarely  if  ever  does  measurement.  An¬ 
swering  all  of  the  questions  typically  takes  about  15  or  20  minutes  at  most,  less  than  5  minutes  if 
your  organization  does  not  customarily  do  measurement.  As  always,  any  information  that  could 
identify  you  or  your  organization  will  be  held  in  strict  confidence  by  the  SEI  under  promise  of 
non  disclosure. 

You  will  find  your  personalized  form  on  the  World  Wide  Web  at 
https://seir.sei. emu. edu/feedback/Measurement2008.asp?ID=<random>. 

Please  be  sure  to  complete  it  at  your  earliest  convenience  —  right  now  if  you  can  make  the  time. 
You  may  save  your  work  at  any  time,  and  you  may  return  to  complete  your  form  over  more  than 
one  session  if  you  wish.  Everything  will  be  encrypted  for  secure  transfer  and  storage. 

Please  feel  free  to  contact  us  at  sei-measurement@sei.cmu.edu  if  you  have  any  questions  or  con¬ 
cerns  about  this  work,  or  have  any  trouble  completing  your  form  over  the  web.  We'll  get  back  to 
you  right  away. 

Thank  you  in  advance. 

Measurement  and  Analysis  Team 
Software  Engineering  Institute 


133  I  CMU/SEI-2008-TR-024 


Facsimile  of  First  Reminder 

Reminder  -  The  State  of  Software  Measurement  Practice  -  2008 
To  the  attention  of:  <Invitee> 

Date:  May  21,  2006 

About  two  weeks  ago,  we  asked  you  to  complete  a  questionnaire  for  the  SETs  third  annual  survey 
about  the  state  of  measurement  and  analysis  practice.  We  have  begun  receiving  responses;  how¬ 
ever  we  have  not  yet  heard  from  you. 

Remember  that  you  are  part  of  a  carefully  selected  sample.  We  need  to  hear  from  you  to  have  ac¬ 
curate  and  useful  results,  whether  or  not  your  organization  is  a  regular  user  of  software  measure¬ 
ment. 

Please  be  sure  to  complete  your  questionnaire  by  June  4.  As  a  reminder,  you  will  find  it  on  the 
World  Wide  Web  at  https://seir.sei. emu. edu/feedback/Measurement2008.asp?ID=<random>. 

It  should  take  you  about  15  or  20  minutes,  less  than  5  minutes  if  your  organization  does  not  cus¬ 
tomarily  do  measurement.  As  always,  your  candid  answers  and  anything  that  could  identify  you 
or  your  organization  will  be  held  in  strict  confidence  by  the  SET  An  early  summary  of  the  results 
will  be  sent  to  those  who  have  completed  their  questionnaires.  We  hope  that  it  will  prove  to  be 
useful  for  you. 

Please  contact  us  at  sei-measurement@sei.cmu.edu  if  you  have  any  questions  or  concerns  about 
this  work,  or  have  any  trouble  completing  your  form  over  the  web.  We'll  get  back  to  you  right 
away. 

Once  again,  thank  you  in  advance. 

Most  sincerely. 

Software  Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
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Facsimile  of  Second  Reminder 

Second  Reminder  -  The  State  of  Software  Measurement  Practice 
To  the  attention  of:  <Invitee> 

Date:  June  4,  2006 

As  you  know,  the  Software  Engineering  Institute  has  been  conducting  its  third  annual  survey 
about  the  state  of  measurement  and  analysis  practice.  Naturally  we  hope  that  the  results  will  help 
you  better  judge  your  own  progress  relative  to  the  successes  and  challenges  reported  by  others. 

We  have  been  encouraged  by  the  number  of  responses  we  have  received  so  far;  however  we  have 
not  yet  received  a  reply  from  you. 

Please  do  make  about  15  or  20  minutes  time  to  complete  your  questionnaire  as  soon  as  you  possi¬ 
bly  can,  by  June  18  if  at  all  possible.  Remember  that  you  may  return  to  complete  it  over  more  than 
one  session  if  that's  easier  for  you. 

And  please  be  sure  to  let  us  know  if  your  organization  does  not  develop  software  or  does  not  cus¬ 
tomarily  do  software  measurement.  Doing  that  should  take  you  well  less  than  5  minutes. 

Once  again,  your  personalized  form  is  on  the  World  Wide  Web  at 
https://seir.sei. emu. edu/feedback/Measurement2008.asp?lD=<random>. 

Thank  you  for  your  contribution  and  cooperation  with  this  effort.  We  hope  that  the  results  will 
prove  to  be  valuable  to  you. 

Most  sincerely. 

Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
sei-measurement@sei.cmu.edu 
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Facsimile  of  Final  Reminder 

Final  Reminder  -  The  State  of  Software  Measurement  Practice 
To  the  attention  of:  <Invitee> 

Date:  June  19,  2006 

We  are  writing  to  you  about  the  Software  Engineering  Institute's  third  annual  survey  about  the 
state  of  measurement  and  analysis  practice.  We  have  not  yet  received  your  completed  question¬ 
naire. 

We  expect  the  results  to  be  of  considerable  value  to  organizations  that  wish  to  continue  improving 
their  measurement  practices  and  their  resulting  performance  outcomes.  Of  course,  your  answers 
are  needed  to  make  the  survey  an  accurate  representation  of  the  state  of  the  practice. 

Please  do  make  time  to  complete  your  questionnaire  as  soon  as  you  possibly  can,  by  the  end  of 
this  month  at  latest.  We  will  let  contributors  know  as  soon  as  the  results  are  ready. 

And  please  be  sure  to  let  us  know  if  your  organization  does  not  develop  software  or  does  not  cus¬ 
tomarily  do  software  measurement.  Doing  that  should  take  you  well  less  than  5  minutes. 

Once  again,  your  personalized  form  is  on  the  World  Wide  Web  at 
https://seir.sei. emu. edu/feedback/Measurement2008.asp?lD=<random>. 

Remember  that  you  may  return  to  complete  it  over  more  than  one  session  if  that's  easier  for  you. 
Of  course,  your  answers  will  be  held  in  strict  confidence. 

Thank  you  again  in  advance  for  your  contribution  and  cooperation  with  this  effort.  We  hope  that 
the  results  will  prove  to  be  valuable  to  you. 

Most  sincerely. 

Measurement  and  Analysis  Team 
Software  Engineering  Institute 
Carnegie  Mellon  University 
sei-measurement@sei.cmu.edu 
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Appendix  C:  Open-Ended  Replies:  Qualitative  Perspectives  on 
the  Quantitative  Results 


This  appendix  contains  the  free-form  textual  answers  to  selected  “open-ended”  questions  from  the 
questionnaire  for  the  survey  of  high  maturity  organizations.  Some  of  the  answers  have  been  edited 
to  for  purposes  of  nondisclosure.  The  questions  include  the  following. 

•  S1Q2:  Which  of  the  following  best  describes  the  role  you  play  in  your  organization?  - 
“Other”  responses  (page  137) 

•  S1Q3:  How  is  your  organization  best  described?  -  “Other”  responses  (page  138) 

•  S1Q4:  What  is  the  primary  focus  of  your  organization’s  work?  -  “Other”  responses  (page 
140) 

•  S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide  (page  140) 

•  S3Q3:  How  much  automated  support  is  available  for  measurement  related  activities  in  your 
organization?  -  “Other”  responses  (page  146) 

•  S4Q4:  Which  of  the  following  (often  interim)  process  performance  outcomes  are  routinely 
predicted  with  process  performance  models  in  your  organization?  -  “Other”  (page  147) 

•  S6Q1:  Following  is  a  series  of  statements  about  the  kinds  of  technical  challenges  that  pro¬ 
jects  sometimes  face.  How  well  do  they  describe  your  organization?  (page  147) 

•  S7Q1:  Which,  if  any,  of  the  following  have  been  major  obstacles  during  your  organization’s 
journey  to  high  maturity?  -  “Other”  responses  (page  147) 

•  S7Q3:  What  have  been  the  greatest  barriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them?  (page  148) 

•  In  Conclusion:  Is  there  anything  else  that  you  would  like  to  tell  us  about  the  use  or  useful¬ 
ness  of  measurement  and  analysis  in  your  organization?  (page  160) 

S1Q2:  Which  of  the  following  best  describes  the  role  you  play  in  your  organization?  - 
“Other”  responses 

I  am  leading  development  processes,  especially  CMMI  activities. 

Former  executive  manager  and  sponsor  in  this  organization 

Software  Functional  Manager  -  responsible  for  software  staff,  tooling,  process,  and  as  a  check  on 
line  (project)  activities 

Business  Excellence  Manager 

QA  Manager 

Engineering  Process  Group  Chairperson 
Chief  Executive  of  the  organization 
Manager  of  EPG  and  QA 
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S1Q2:  Which  of  the  following  best  describes  the  role  you  play  in  your  organization?  - 
“Other”  r^onses 

consultant,  past  VP 

Director 

CMMI  compliance  lead,  measurements  lead,  backup  to  the  SEPG  lead  and  backup  to  the  CMMI 
assessment  project  lead 

Managing  director  of  [company  name] 

Enterprise  measurement  IPT  lead 

SEPG  leader 

Lead  appraiser  who  performed  the  last  appraisal 

Process  consultant 

SEPG 

Manager  of  division  measurement  and  analysis  office  and  high  matnrity  initiatives 

In  charge  of  process  management 

Software  qnality  engineering  manager;  organization  process  gronp  (a.k.a.  SEPG)  lead 

S1Q3:  How  is  your  oi^anization  best  described?  -  “Other”  rei^nses 

Shrink-wrap  or  custom  installation  of  enterprise  solutions  and  contracted  new  development 

Software  development  &  maintenance 

Development  &  maintenance  services 

Global  provider  of  software  solutions  and  IT  services,  ranging  from  specialization  in  quality  cus¬ 
tom  software  development,  prodnct  development,  consulting  services  to  systems  integration,  and 
outsonrcing 

System  integrator 

Financial  institntion 

Commercial  contractor 

Design  &  engineering  consultancy 

Product  and  technology  development  for  aerospace  and  automation  control  systems 

Custom  development  applications,  device  driver  applications,  COTS  with  and  without  customiza¬ 
tion  and  application  maintenance 

An  IT  company  that  delivers  consulting,  systems  integration,  and  outsourcing  solutions  to  cus¬ 
tomers 

Software  solutions  and  services  provider  for  application  development  and  maintenance 
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S103:  How  is  lour  om^^tion  best  described?  -  “Other”  moonses 

Government  contractor  for  several  government  departments 

All  the  above 

Software  solutions  and  services  provider 

Software  development  and  integration,  focused  on:  IT  consulting,  business  intelligence,  develop¬ 
ment  and  integration,  software  factory,  security,  testing  factory,  and  consulting  services 

Software  products  and  projects  related  to  scientific,  engineering,  financial,  business  analytics  & 
enterprise  applications;  software  product  development 

Service  provision 

Application  software  and  service  provider  for  trade  customers  (development,  maintenance,  test¬ 
ing,  application  management  services 

IT  software  services  organization  with  capabilities  in  application  development  and  COTS  imple¬ 
mentation 

Software  development  company  providing  solutions  to  the  industry.  Also  defense  subcontractor. 

Offshore  software  service  provider 

IT  services  provider 

Software  factories  for  external  customers 

Defense  contractor  and  other  government  contractor 

Software  development  and  maintenance  services 

A  leading  global  provider  of  IT  services  and  business  solutions  for  software  development,  main¬ 
tenance,  enhancement,  testing  and  production  support 

COTS  and  contracted  new  development 

*  Implementation,  Maintenance  &  Support  of  COTS  products 

*  Development,  maintenance  &  support  using  [specified  commercial]  technologies 

Item  number  1 ,2  and  3  above 

Application  management  services,  including  new  development/customization  and  on-going  main¬ 
tenance  of  self-developed  or  off-the-shelf  packages  for  both  in-house  and  external  clients 

Software  vendor  (insurance  enterprise  solutions) 

A  mixture  of  in-house  or  proprietary  development  or  maintenance  and  contracted  new  develop¬ 
ment 

Provider  of  IT  solutions  and  services.  Infrastructure  services,  engineering  and  industrial  services, 
asset  based  solutions,  consulting 

Product  development/implementation  and  contracted  new  development  and  maintenance 
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S1Q4:  What  is  the  primaiiy  focus  of  your  ot^^mtetion’s  work?  -  “Other”  responses 

Systems  development  &  maintenance 

Development  &  maintenance  services 

IT  solutions  including  new  application  development,  maintenance  and  testing. 

Product  development,  customized  solution  development.  Maintenance  services,  &  IT  related  ser¬ 
vices 

Product  and  system  development,  maintenance  and  sustainment 

Application  development,  product  implementation,  maintenance  and  sustainment 

Application  development,  maintenance,  production  support,  product  engineering,  system  integra¬ 
tion,  IT  consultancy,  package  implementation,  IT  Infrastructure  management,  product  develop¬ 
ment,  testing  and  validation  services 

Maintenance  and  sustainment 

All  the  above 

IT  consulting,  business  intelligence,  development  and  integration,  software  factory,  security,  test¬ 
ing  factory,  and  consulting  services 

System  development  and  maintenance 

Both  product  and  system  development  and  maintenance 

Design,  development  and  maintenance  of  software 

Software  development  and  maintenance  services 

Software  development,  maintenance,  enhancement,  testing,  production  support  and  package  im¬ 
plementation 

Software  development,  application  maintenance  and  infrastructure  management 

Product  and  system  development,  maintenance  and  sustainment 

both  item  1  and  2 

In-house,  maintenance  and  sustainment 

Development  and  maintenance  for  external  clients 

Engineering,  operations  and  maintenance 

IT  solutions  and  services,  infrastructure  services,  engineering  and  industrial  services 

S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide? 

Role  based  process  training  is  conducted  periodically  for  the  project  managers.  Metrics  council  mem¬ 
bers  are  provided  training  on  measurements  related  areas  by  both  internal  and  external  faculty. 

Monthly  reviews  of  measurement  with  executive  management 
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S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide? 

On  the  job  training  by  experienced  people  to  people  who  are  responsible  for  measurement 

Quality  coaches  for  knowledge  transfer  to  the  projects 

On  the  job  training  from  specialists  to  process  practitioners  within  6  sigma  projects 

Specialized  library  of  books  available  for  consultation 

Our  activity  is  supported  by  measurement-related-specialists  in  another  division  within  our  com¬ 
pany. 

Lean  and  six  sigma  training 

Training  and  application  of  training  by  measurement  champions  and  mentoring  customers  on  belt 
and  belt-like  activities.. 

Lean  Six  Sigma  and  Theory  of  Constraints  training 

Statistical  Process  Control 

Training  in  COTs  Tools 

Measurement  workshops  for  project  managers  and  quality  managers. 

Statistics  courses  in  local  universities. 

Six  Sigma  black  belt  programs,  Green  belt  programs  and  introduction  to  Measurement  techniques 
to  new  engineers.  Refresher  courses  to  the  managers  and  project  engineers 

"Manage  project  by  data"  is  trained  for  Project  managers  and  QA  staff. 

Training  on  use  of  the  [organization’s  measurement  repository]  to  collect  data  is  provided  for  all 
staff 

Six  Sigma,  PBM,  EVM, 

Statistical  methodology 

Basic  Statistics 

Six  Sigma  Green  Belt  (self  paced  -  online  training) 

We  hire  (temporarily)  a  specialized  consultant  to  help  on  the  performance  model  building  activi¬ 
ties,  as  needed.  These  consultants  act  like  mentors  or  coaches  for  the  model  builders  and  main- 
tainers. 

1 .  Internal  facilitation  on  basic  statistics 

2.  Internally  developed  Six  Sigma  Green  Belt  training. 

Statistical  Process  Control 

We  provide  training  on  usage  of  statistical  techniques.  Six  Sigma  Yellow  Belt  /  Green  Belt  / 

Black  Belt  training.  Additionally  role-based  training  is  provided  on  applicability  and  usage  of 
metrics  for  statistically  managing  the  projects. 

Open  contracts  with  measurement  specialists,  specifically  statistics  specialists. 

Training  on  software  measurement  program  that  covers  measurement  related  activities  &  artifacts 
both  at  organization  level  as  well  as  project  level. 

One-on-one  mentoring  between  the  organization  quality  engineer  and  the  project  quality  engineer. 

We  provide  a  wide  variety  of  measurement  training.  A  large  majority  of  employees  have  either  6- 
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S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide? 

Sigma  Black  Belt  Expert  training  or  6-Sigma  Green  Belt  training.  In  addition,  we  have  a  2-week 
Program  Management  course  that  all  PMs  must  attend  which  has  significant  measurement  train¬ 
ing  included.  These  PMs  are  tested  prior  to  the  course  and  after  the  course  to  ensure  the  required 
skills/knowledge  are  obtained.  In  addition  a  variety  of  internal  measurement  training  is  developed 
by  6-Sigma  Black  Belts  and  delivered  to  employees  at  all  levels.  All  managers  receive  the  3  day 
CMMI  training.  Several  selected  process  engineers  have  attended  the  Understanding  High  Matur¬ 
ity  Practices  course  offered  by  the  SET  We  also  offer  all  employees  a  Work  Study  program  which 
allows  them  to  focus  on  their  items  of  interest  -  many  work  study  groups  have  been  focused  on 
measurement  and  analysis. 

Basic  concepts  of  measurement 

Eunction  Point  counting 

Ongoing  facilitation  to  projects 

Regular  skill  upgrades  through  in-house  training  sessions. 

Measurement  specialists  work  one-on-one  with  project  analysts. 

1 .  SPC  techniques 

2.  Training  on  measurement  tools 

Metrics  based  management  as  part  of  engineering  process  training 

Coaching  and  mentoring  by  Six  Sigma  black  belts  in  the  organization 

We  provide  to  our  internal  employees  involved  in  metrics  activities  training  from  [external  train¬ 
ing  organizations]  for  helping  us  in  our  six  sigma  projects  &  building  process  performance  mod¬ 
els. 

GQM 

SPC  work  shop  and  case  studies 

a)  Training  on  knowledge,  usage  and  practices  of  metrics  at  [external  training  agency] 

b)  Training  on  statistical  tools 

c)  Training  on  [project  management  Tool]  for  capturing,  usage,  practice  and  reporting  of  meas¬ 
urement  data 

d)  Training  in  Eunction  Point  and  Use  Case  Point  methodology  for  estimation  of  efforts,  cost  etc. 

e)  Eacilitation  to  project  and  support  function  users  with  respect  of  measurement  related  activities 

We  have  hired  a  senior  professor  from  [university]  as  a  consultant  to  train  our  folks  on  statistical 
analysis  and  set  direction  for  our  metrics  program 

Regular  training  on  [organization’s  quality  manual] 

LEAN,  Process  Mapping  /  Re-engineering,  VSM,  6  Sigma 

Project  risk 

Consulting  services  to  mentor  the  users  of  the  data  and  internal  coaches  and  mentors,  tailored 
specifically  to  their  project  work 

-  SAV  process  improvement  training  on  measure  and  analysis 

-  Statistical  Process  Control  training 

-  Inspection  and  defects  training 

-  SW  project  quality  and  measurement  mentoring  (face  to  face) 
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S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide? 

-  SW  project  measurement  mentoring  [using  specified  management  system] 

-  Quality  audit  and  mentoring  (face  to  face),  etc. 

SPC 

Regular  workshops  are  conducted,  on  an  ongoing  basis  for  project  managers  and  middle  manag¬ 
ers  to  brief  them  and  bring  them  up  date  with  latest  requirements  of  CMMI 

Our  organization  provides  the  training  of  measurement  tools 

Statistical  Process  Control,  basic  statistics,  process  performance  model  Course,  CMMI  level  5 
training. 

Lean  facilitation  and  training  provided  both  internally  and  through  vendor  organizations 

Statistical  Process  Control  training  (internally) 

Quantitative  management  overview 

Regular  refresher  programs 

Every  individual  technical/non-technical  person  is  trained  to  collect  his/her  own  data. 

CMMI  Lead  Appraisers  and  measurements  matter  experts  provide  regular  briefing  sessions  to  all 
levels  of  the  organization.  In  addition  each  project  manager  is  assigned  a  quality  analyst  to  assist 

Our  normal  [process]  provides  continuous  training  reinforcement  via  the  team  meetings. 

Specific  training  about  the  models  developed  and  to  be  used  in  the  organization  is  provided  on  at 
least  an  annual  basis  to  all  measurements  users  (or  when  models  change). 

Measurement  training  for  an  as-needed  basis  for  various  organizational  roles. 

-  Internal  1-day  Measurements  Analysis  class  as  a  basic  for  all  process  and  metrics  staff  who  col¬ 
lect  and  manage  the  baseline  data,  and  build  and  maintain  the  model. 

-  Internal  brief  sessions  about  how  to  use  measurements  as  well  as  embedding  it  in  some  current 
courses,  such  as  a  metrics  brief  session  for  project  managers,  and  how  to  use  metrics  within  peer 
review  moderator  course. 

-  Green  Belt  Six  Sigma  by  external  trainer 

Six  Sigma  Green  Belt  training 

Brown  bag  sessions 

1 .  Green  belt  training 

2.  Lean  sigma  training 

3.  High  Maturity  workshop  for  Managers 

4.  Advanced  metrics  training 

5.  Statistical  process  control 

Six  Sigma  MBBs  and  Lead  Black  belts  mentor  the  analytics  team 

We  have  developed  internal  training  courses  that  are  conducted  at  regular  intervals,  for  the  differ¬ 
ent  roles  in  the  organization.  The  Six  Sigma  program  is  designed  to  provide  middle  managers  and 
senior  managers  a  good  understanding  of  measurement  and  analysis  using  different  statistical 
techniques. 

Case  study  based  metrics  training  program  provided  for  project  managers  and  leads 

PMP  certification  program  for  project  managers 
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S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide? 

Process  Performance  Model  builders  are  encouraged  to  attend  SPIN  meetings  and  to  do  extensive 
literature  search. 

We  bring  in  consultants  conversant  with  high-maturity  practices  to  perform  customized  training 
and  workshops. 

Contact  with  a  specialized  mentor  and  conferences  as  applicable 

Process  workshops  and  quizzes 

Measurement  &  analysis,  Metrics,  SPC,  SQC 

SEI  consultant 

The  PPM  training  session  is  recorded  and  available  on  the  [company]  intranet  for  all  users  to  ac¬ 
cess  anytime. 

SEPG  process  consultants  attached  to  every  project  are  available  for  mentoring  throughout  the 
duration  of  the  project. 

Green  Belt  and  Black  Belt  Training,  CMMI  High  Maturity  Mentoring  and  special  topics  sessions 

Estimation  training,  metrics  monitoring  and  tracking  tool  training  and  other  software  process  and 
tool  training 

-  Customer  specific  measurement  and  tools  training 

-  Standard  training  on  data  capturing  for  practitioners  and  QA 

-  Metrics  management  training  for  project  leads  and  QA  people 

-  Senior  management  review  for  metrics  analysis 

QPM  CBT 

Kickoffs  w/  new  programs  to  set  expectations  for  program  measurement  responsible  individuals 
(PMRIs).  PMRIs  are  mentored  by  members  of  the  Enterprise  Measurement  IPT  as  part  of  pro¬ 
gram  rollout. 

We  have  an  excellent  tuition  reimbursement  program  that  supports  staff  pursuit  of  graduate  edu¬ 
cation.  Many  of  our  staff  take  advantage  of  this.  Our  primary  model  builder  and  maintainer  re¬ 
ceived  formal  education  in  statistical  techniques  through  this  program.  Our  process  improvement 
program  has  benefited  significantly  through  this  formal  education  avenue. 

SPC 

Weekly  meetings  of  the  measurement  team 

We  have  a  measurement  and  analysis  training  course  on  line,  we  also  have  instructor-led  6  Sigma 
training  courses 

Quality  measures,  customer  satisfaction  measures,  performance  measures 

Six  Sigma  Green  Belt  and  Black  Belt  training  and  mentoring;  CMMI  Level  4  &  5  train¬ 
ing/mentoring  (in-house);  and  SEI  High  Maturity  training 

We  provide  in-house  training  on  Statistical  Project  Management  and  Quantitative  Process  Meas¬ 
urement.  Apart  from  that  there  is  a  very  strong  mentoring  that  happens  from  the  senior  manage¬ 
ment  on  specific  areas. 

Black  Belts  work  with  Green  Belts.  Belts  are  'seeded'  around  the  organization  so  that  knowledge 
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S2Q4:  What,  if  any,  other  types  of  measurement  related  training  or  mentoring  does  your 
organization  provide? 

is  shared.  The  Metrics  Office  also  provides  internal  coaching  and  mentoring. 

Statistical  concepts,  Six  Sigma,  Quantitative  Management 

There  are  2  levels  of  trainings  we  focus  upon 

Level  1:  e.g..  Six  Sigma  for  continuous  improvements  with  respect  to  measurement  and  it's  actual 
benefit  to  the  organizations  goals  and  objectives 

Level  2:  e.g.,  ISO  20K  ,  IT  Service  Management  to  strengthen  delivery  of  ALL  our  services  to  the 
customer 

Six  Sigma  Black  Belt  Training 

In  our  organization  exists  an  area  that  generates  the  baselines  and  process  performance  models, 
this  area  is  performed  by  Green  Belts  and  Black  Belts  and  they  support  the  projects  with  coaching 
about  metrics  interpretation  and  the  use  of  some  statistics  tools. 

Another  type  of  mentoring  is  provided  by  the  improvement  teams  (IM).  The  IMs  are  groups  of 
professionals  that  work  in  some  subprocess,  they  have  improvement  meetings  where  they  analyze 
and  understand  the  process  performance  behavior,  with  meeting  participation  by  a  Black  Belt  or 
Green  Belt  that  supports  the  IM  to  understand  and  apply  statistics  in  order  to  find  a  root  cause  for 
process  variation. 

Before  the  SCAMPI  our  Lead  Assessor  provides  the  SCAMPI  Team  a  lot  of  Maturity  Level  4  and 
5  materials  in  order  to  have  a  better  understanding  about  metrics,  this  material  had  been  used  like 
reference  in  some  initiatives  and  understanding. 

A  variety  of  courses  e.g.,  QMS  training,  performance  engineering;  six  sigma,  professional  mem¬ 
berships,  magazine  subscriptions,  certifications  are  sponsored  for  all  associates  who  wish  to  study 
in  this  area;  books  and  training  are  also  freely  available  for  all  those  who  are  interested 

Explanation  of  metrics  and  measurement  related  to  the  organization  on  a  one  to  one  basis  by  the 
SQA  group  members 

(a)  Working  groups  in  QPM,  CAR  and  M&A  (meeting  every  2-3  weeks)  mentor  and  guide 
launching  and  optimizing  activities  in  these  areas  under  direction  of  experts. 

(b)  Lean  Six  Sigma  training  for  Green  Belts  (2  weeks)  and  Black  Belts  (4  weeks),  provide  men¬ 
toring  and  guidance  from  Master  Black  Belts.  Numerous  Six  Sigma  projects,  conducted  to  meas¬ 
urably  improve,  are  under  mentoring  of  Black  Belts  and  MBB. 

Periodic  workshops  on  Saturdays,  once  per  month  and  OJT 

QPM  Course 

Statistics  (General,  SPC  and  variance)  courses  (few  levels)  provided  by  the  [organization] 

We  have  internally  developed  statistical  analysis  training  courses;  how  to  develop  the  correct 
measures  courses;  six-sigma  mentors. 

In-house  training  coursed  developed  for  Managing  Process  Performance,  Measurement  for  Data 
Collectors  and  Analysts,  Measurement  Overview,  Measurement  Program:  Orientation  and  Meas¬ 
urement  Workshops. 

Consulting  services  (e.g.,  [specified  consultancy]) 

Statistical  process  control 
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We  encourage  and  sponsor  our  middle  level  managers  to  go  in  for  professional  certifications  in 
the  project  management  realm.  The  course  curriculum  /  certification  requirements  cover  schedul¬ 
ing  and  metrics  measurement,  among  others. 


S3Q3:  How  much  automated  support  is  available  for  measurement  related  activities  in  your 
organization?  -  “Other”  responses 

Macros  developed  in-house  to  collect  verification/validation  data 

Macros  developed  in-house  to  download  data  from  OMR  into  the  PPMs 

Measurement  tool  developed  in-house  to  consolidate  data  from  different  data  sources  ([COTS 
products],  Checklists...) 

We  have  a  proprietary  tool  which  helps  for  complete  comprehensive  project  management,  which  in¬ 
cludes  data  management,  quantitative  data  analysis  on  a  periodic  basis  etc... 

Data  collection  but  not  automated  ([specific  COTS  product]) 

In-house  tools 

Home-made  tool  and  extensive  use  of  open  source  software. 

In-house  developed  tools  provide  data  capture,  graphing,  and  report  capability,  which  can  be 
automated,  batched,  and  scheduled.  This  reduces  these  time-consuming  tasks,  and  the  analysts 
can  focus  on  the  analysis. 

[Commercial  statistics  package]  -  Organization  wide  license  available  for  all  in  the  organization 

[Organization]  has  its  own  web  based  project  management  tool  for  managing  project  management 
activities  including  measurement  activities  and  analysis.  This  tool  generates  various  reports  in¬ 
cluding  reports  on  measurements. 

[Organization]  owned  -  [COTS]  statistical  tool,  [and  other  COTS  tools] 

Proprietary,  best  in  class  manufacturing  and  development  tool  sets 

Process  simulation  model,  [named  COTS  tool 

Self-built 

Apart  from  the  above,  we  have  developed  our  internal  measurement  and  analysis  systems.  These 
intranet  applications  are  extensively  used  for  data  capture,  consolidation,  analysis  and  reporting. 

We  have  in-house  developed  tools  to  carry  out  various  functions  which  can  generate  data  and  re¬ 
ports;  a  request  management  system  tracks  all  the  defects  from  customer  requirements  through 
delivery. 

[Proprietary  time  management  product]  for  data  on  productivity  also  helps  in  determining  esti¬ 
mated  versus  actual  effort 
[Proprietary  testing  product] 

[Proprietary  management  system  product]  which  is  being  further  developed  to  meet  quality  needs 

A  skills  database  for  employee  training  needs  and  skills  and  knowledge  repository  helps  to  pro¬ 
vide  a  lot  of  data  for  organizational  training 

[Proprietary  version  management  product] 

Project  Management  Software 
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S3Q3:  How  much  automated  support  is  available  for  measurement  related  activities  in  your 
organization?  -  “Other”  responses _ 

We  are  also  planning  to  develop  software  that  can  cater  to  the  needs  of  CMMI  practices  while 
integrating  with  already  existing  software  in  [proprietary  product  suite]  mentioned  above. 

A  process  automation  tool  is  under  pilot  stage. 


S4Q4:  Which  of  the  following  (often  interim)  process  performance  outcomes  are  routinely 
predicted  with  process  performance  models  in  your  organization?  -  “Other” 

Support  service  quality  &  productivity 

NOTE:  Performance  models  may  not  meet  the  SEI  essential  characteristics  definition  in  all  cases. 


S6Q1:  Following  is  a  series  of  statements  about  the  kinds  of  technical  challenges  that  pro¬ 
jects  sometimes  face.  How  well  do  they  describe  your  organization? 

Insufficient  schedule 

We  have  a  well-defined  competency  framework  program  to  address  skill  set  gaps  in  [organiza¬ 
tion’s  domain],  technology  and  process. 


S7Q1:  Which,  if  any,  of  the  following  have  been  major  obstacles  during  your  organization’s 
journey  to  high  maturity?  -  “Other”  responses 

In  summary  the  difficulties  arise  due  to: 

Variations  in  governance  models  for  different  customers.  This  results  in  same  type  of  service  hav¬ 
ing  different  project  priorities  and  thus  different  measurement  plans. 

Projects  that  do  not  follow  the  classical  application  development  cycle  (i.e.,  support,  maintenance 
and  enhancement  projects). 

Projects  that  deliver  a  ‘bundle’  of  services  such  as  support  and  enhancements  by  using  the  same 
resources  flexibly 

Need  to  train  more  people  in  modeling  and  using  data  effectively  to  predict  outcomes 
Lesser  Instance  of  repeated  data  of  similar  type,  on  projects  due  to  size  of  the  organization 
Lack  of  enough  data  for  analysis 

Customers  do  not  value  the  activity  and  do  not  wish  to  pay  the  premium  for  high  quality.  Moving 
beyond  CMMI-5  is  NOT  a  desired  state  for  our  business  any  longer  and  is  seen  as  a  negative  due 
to  high  cost.  As  a  result  we  have  made  a  systematic  decision  to  STOP  advertising  our  more  recent 
CMMI  5  assessments  to  cost  sensitive  customers. 

A  high  percentage  of  our  business  is  small  project  maintenance  and  operations.  Little  examples 
exist  in  the  industry  to  leverage  from  for  this  type  of  project.  Statistical  Analysis  and  modeling 
can  be  high  overhead  for  smaller  projects  unless  done  in  a  scaled  approach.  CMMI  model  does 
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S7Q1:  Which,  if  any,  of  the  following  have  been  major  obstacles  during  your  organization’s 
journey  to  high  maturity?  -  “Other”  responses 

not  allow  for  this  scaled  approach  very  well  -  it  is  geared  more  towards  the  large  project  with 
dedicated  measurements  staff. 

-  The  lack  of  automated  project  and  service  data  collection  means  that  we  cannot  provide  real¬ 
time  metrics  to  the  projects. 

-  The  varied  nature  of  existing  and  new  domains  in  the  software  development  center  creates  a 
need  to  change  some  metrics  definitions  and  design  new  metrics.  There  is  not  enough  stable- 
process  historical  data  for  process  performance  models. 

-  The  limitation  of  process  and  metrics  engineers’  effort  and  budget  make  an  insufficient  metrics 
training  and  communication  for  lOOO-i-  staff  (project  managers,  software  development  and  service 
engineers). 

In  a  few  cases,  though  process  performance  measures  are  consistent  within  a  project,  it  has  been 
difficult  to  get  the  consistency  across  projects. 

We  see  a  business  case  need  to  balance  the  approach  as  it  relates  to  Level  4  activities  &  the  prac¬ 
tical  implementation  of  modeling  &  predictive  performance  on  a  broader  scale  than  just  those 
measures  that  are  being  statistically  managed. 

There  are  no  major  obstacles.  Senior  management  is  highly  supportive  and  the  team  has  a  high 
spirit  to  go  for  high  maturity. 

We  failed  to  demonstrate  the  leverage  measurement  and  analysis  provides  for  business  process 
improvement. 

Because  this  is  mainly  a  service  environment  some  of  the  major  processes  are  usually  tailored  at 
the  account  level.  This  means  that  it  is  difficult  to  aggregate  them  across  the  organization. 

Resistance  to  new  improvements  that  brings  change  still  exists,  but  it  is  coming  to  be  accepted  as 
a  normal  culture. 

Providing  digitization  support  in  a  timely  manner  is  a  challenge. 


S7Q3:  What  have  been  the  greatest  barriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them? 

Convincing  the  project  team  members  that  process  modeling  and  adherence  to  processes  will  re¬ 
sult  in  better  quality  and  cost  control. 

Application  of  high  maturity  models  meant  for  large  projects  to  small  projects  of  <  50  engineer¬ 
ing-months  of  effort. 

We  don't  have  enough  methodology  to  control  our  products’  quality. 

Therefore  we  are  collecting  various  techniques  and  trying  it. 

To  convince  managers  that  quantitative  management  will  improve  their  management  activities, 
and  will  improve  the  project  performance. 

It  takes  a  long  time  to  convince  managers;  we  first  convinced  the  senior  management,  then  con¬ 
tinuous  training  to  managers  with  good  examples  from  other  organizations. 

The  greatest  barrier  has  been  the  knowledge  needed  to  get  the  "gestalt"  understanding  of  the 
model;  once  the  problem  was  understood,  the  solution  was  clear.  In  order  to  overcome  this  bar¬ 
rier,  extensive  time  was  spent  by  a  few  key  people  to  acquire  the  knowledge  by  getting  consulta- 
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S7Q3:  What  have  been  the  greatest  harriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them? 

tion  from  experts,  training,  reading  books  and  papers,  and  experimenting  the  acquired  knowledge 
in  real  projects. 

People  buy-in  and  involvement.  We  overcame  them  by  effective  communication,  training,  change 
management  and  setting  up  rewards  &  recognition  measures. 

In  an  old,  long  service  organization  that  takes  pride  in  the  diversity  of  its  work,  we  had  difficulty 
shifting  from  long  period  to  short  period  measurements  because  it  just  wasn't  done  that  way.  We 
developed  a  model  with  a  modest  number  of  standard  work  package  types  with  reusable  meas¬ 
urement  patterns.  This  reduced  training  load,  improved  the  consistency  and  amount  of  relevant 
data. 

The  combination  of  Six  Sigma  with  CMM/CMMI  Level  3,  4,  &  5  has  been  very  helpful.  It  broad¬ 
ens  the  corporate  expectation  of  objectivity  and  useful  measurement  as  well  as  provides  a  useful 
toolkit. 

The  fundamental  nature  of  life  is  to  balance  the  need  to  change  (improve)  with  the  need  for  stabil¬ 
ity  (control).  We  and  many  others  have  had  problems  to  the  effect  that  the  organization  would 
rather  control  than  improve.  Many  managers  and  others  are  not  worried  if  an  organization  is  "in 
control"  but  not  improving.  Given  an  impetus  from  ISO  or  CMMI,  and  maybe  with  an  inclination 
to  let  "perfect"  kill  "better,"  and  a  preference  for  control  over  real  improvement,  without  care,  the 
CMMI  presents  opportunities  to  straight-jacket  organizations  in  perfectly  executed  but  relatively 
meaningless  minutia.  It  is  important  to  let  noise  (diversity)  into  the  system,  to  think  about  alterna¬ 
tives,  to  keep  the  end  goals,  but  let  the  approaches  wander.  Support  the  objective  thinkers  in  your 
organization.  If  you  don't  have  enough  staff  who  are  "naturals"  at  this,  people  who  hate  the  de¬ 
scription,  "out  of  the  box  thinkers,"  consider  training  some  staff  in  Altschuler's  TRIZ. 

Sufficient  data  points  for  creation  of  performance  baselines  and  models  and  then  convincing  the 
managers  to  use  the  baselines.  This  was  overcome  by  ensuring  management  commitment  and 
conducting  awareness  session  on  the  topic. 

Changing  from  a  culture  of  "deliver  on  deadline  regardless  of  completion  state"  to  "deliver  only 
when  quality  standards  are  met." 

We  have  difficulties  in  change  management  in  the  continuous  growth  of  the  organization  size. 
Dealing  with  these,  we  built  an  in-house  management  suite  to  transfer  information  all  over  the 
organization. 

Barriers  -  Lack  of  confidence  that  the  cost  of  moving  to  higher  maturity  levels  is  worth  the  bene¬ 
fit.  Still  have  skepticism  in  parts  of  the  organization. 

Actions  -  Showing  the  benefit  in  data  from  various  parts  of  the  organization  has  helped  to  miti¬ 
gate  the  skepticism 

Resource  constraints 

Barrier:  The  insufficient  understanding  on  PPM  model. 

Overcome  Method:  Our  LA  of  CMMI  level  5  appraisal  provided  consultation  on  this  area. 

We  took  pains  most  to  understand  statistics.  We  take  external  seminars  on  statistical  nnderstand- 
ing,  and  are  spreading  the  concept  in  our  company. 

We  misinterpreted  some  high  level  practices  and  defined  some  subprocesses  that  were  not  imple¬ 
mented  as  required  by  CMMI.  It  was  not  caught  in  one  of  the  performed  assessments  (SCAMPI 

A! !).  As  a  consequence,  we  used  the  model  in  an  improper  way  during  almost  a  year.  After  the 
issue  was  detected  (unfortunately  during  another  SCAMPI  A  evaluation)  we  hired  a  specialized 
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to  high  maturity?  What  have  you  done  to  overcome  them? _ 

consultant  to  help  us  to  re-interpret  the  practices  and  to  redesign  these  subprocesses.  After  the 
appropriate  internal  communication  and  (re)training,  we  consider  that  the  mistake  has  been  cor¬ 
rected. 

The  difficulties  have  arisen  due  to: 

1)  Variations  in  governance  models  for  different  customers.  This  results  in  the  same  type  of  ser¬ 
vice  having  different  project  priorities  and  thus  different  measurement  plans. 

2)  Projects  that  do  not  follow  the  classical  application  development  cycle  i.e.  support,  mainte¬ 
nance  and  enhancement  projects) 

3)  Projects  that  deliver  a  “bundle”  of  services  such  as  support  and  enhancements  by  using  the 
same  resources  flexibly 

To  overcome  some  of  these  issues,  we  have 

1)  Created  service  specific  metrics  definitions 

2)  Decentralized  the  metrics  office  from  the  corporate  level  to  individual  business  units  for  better 
understanding,  mentoring  and  facilitation  at  the  project  level.  The  corporate  metrics  office  still 
exists  for  preparing  PPMs  and  PCBs,  the  team  is  headed  by  a  person  with  college  degree  in  statis¬ 
tics. 

3)  Initiated  a  series  of  project  trainings  on  basic  statistics 

The  greatest  barrier  is  the  variety  of  existing  and  new  domains  in  the  organization,  and  their 
growth.  We  have  been  trying  to  collect  and  display  metrics  for  each  set  for  each  domain. 

-Number  of  data  points  is  insufficient  to  build  Process  Performance  Models 

-Data  accuracy  is  not  there 

-Statistical  domain  expertise  is  not  there 

-Data  collection  mechanism  is  weak 

Ensuring  awareness  and  consistent  usage  of  high  maturity  practices  across  the  organization,  con¬ 
sidering  the  rapid  growth  and  large  number  of  new  recruitments  was  the  biggest  challenge.  This 
risk  was  overcome  by  developing  e-learning  materials  for  implementing  high  maturity  practices, 
training  coverage,  continuous  process  facilitation,  communications,  focused  audits  on  high  matur¬ 
ity  practices,  conducting  quiz  and  other  promotional  events  to  spread  awareness  of  high  maturity 
practices. 

The  biggest  barrier  has  been  organizational  culture.  We  used  to  think  doing  software  was  art,  not 
an  industry.  We  have  evolved,  from  an  art  production  to  an  industry  production,  managing  our 
software  development  center  based  on  models  such  as  CMMI  and  ISO  9001:2000. 

Development  of  Process  Performance  Models  is  not  that  easy. 

To  convince  middle  management  to  spend  extra  time  on  project  management  issues,  including 
measurements  and  modeling. 

Change  in  behavior 

It  was  hard  to  identify  stable  processes  because  it  was  hard  to  find  out  the  real  driving  parameter 
that  we  now  call  [named  parameter].  Once  it  was  identified,  we  found  that  many  processes  are 
dependent  on  this  driving  parameter. 

Understanding  what  a  process  performance  model  is  and  how/when  it  can/should  be  used. 
Changing  the  culture  from  a  "Seat  of  the  Pants"  cowboy  driven  environment  to  one  that  utilizes 
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measurement  data  to  make  or  support  decisions.  Primarily  it  was  overcome  by  showing  historical 
measurement  data  where  the  measurement  program  had  identified  issues  prior  to  their  otherwise 
becoming  visible  to  management.  Also  we  had  true  executive  management  buy-in.  We  estab¬ 
lished  our  program  and  successfully  demonstrated  level  5  capabilities,  but  now  we  must  continue 
to  expand,  refine  and  strengthen  the  capability.  Executive  buy-in  is  a  major  factor  and  lack  of  it  is 
the  quickest  way  to  fail 

The  institutionalization  of  the  processes  to  support  the  activities  of  high-maturity,  associated  with 
the  culture  of  the  organization.  To  remove  this  block  we  used  examples  and  results  of  improve¬ 
ment  in  some  projects  with  the  use  of  those  practices. 

People  feel  insecure  applying  new  knowledge  they  do  not  fully  understand  or  control.  A  lot  of 
training  is  required. 

Lack  of  previous  expertise  in  the  subject.  We  have  collected  all  the  reference  books  in  the  market 
and  hired  external  training  and  support  from  the  European  Software  Institute. 

Having  project’s  identify  process  improvements.  We  have  increased  our  partnering  relationships 
with  the  projects  and  have  provided  updates  to  our  process  documentation  for  recommended  im¬ 
provements  that  should  be  flowed  up  to  the  integrated  process  group  for  evaluation  and  piloting  if 
the  improvement  is  deemed  to  support  lean  process  execution  or  shows  the  improved  process  to 
be  more  cost  effective. 

The  two  greatest  barriers  faced  by  the  organization  were: 

1 .  Conceptualization  of  process  performance  models  suited  to  a  project's  context  and 

2.  Training  the  practitioners  on  usage  and  sustenance  of  the  high  maturity  practices. 

The  first  barrier  posed  challenges  since  projects  in  the  organization  were  disparate  in  nature  and 
one  model  would  not  have  suited  all  the  projects.  This  meant  rolling  out  generic  process  perform¬ 
ance  models  at  an  organization  level,  and  these  were  tailored  to  suit  a  project’s  needs.  There  were 
many  brainstorming  sessions  held  among  the  [modeling  team]  members  and  the  delivery  team  in 
the  model  formulation.  The  delivery  team  provided  inputs  in  terms  of  possible  alternatives  for 
process  composition  and  the  [modeling  team]  members  helped  in  modeling.  Through  constant 
interaction,  project  specific  models  were  rolled  out,  and  the  same  were  reviewed  by  senior  man¬ 
agement  in  the  monthly  review  meetings. 

The  second  barrier  posed  challenges  in  terms  of  getting  the  intent  of  high  maturity  practices  to  the 
practitioners.  "Statistical  thinking"  was  needed  for  the  journey,  and  multiple  programs  were 
launched  to  achieve  the  same.  Along  with  the  project  managers,  "process  champions"  were  identi¬ 
fied  in  all  projects.  Waves  of  High  Maturity  workshops  were  conducted  in  the  organization  to  get 
the  practitioners  to  think  statistically.  There  were  many  quizzes,  and  awareness  mailers  were  sent 
out  periodically.  Projects  were  rewarded  for  exceptional  implementation  of  high  maturity  prac¬ 
tices.  Applications  of  basic  and  advanced  statistical  techniques  in  software  project  management 
were  discussed  with  practitioners  periodically.  Also  knowledge  sharing  by  projects  on  high  ma¬ 
turity  practices  was  conducted  regularly. 

Thinking  in  terms  of  statistical  performance  is  a  paradigm  shift  from  "planned  versus  actual" 
thinking.  The  following  have  helped:  enhancing  tools  to  automate  analysis;  providing  support  to 
projects  for  analysis;  conducting  quarterly  reviews  for  projects  to  share  their  activities  and  im¬ 
provements. 

In  order  to  maintain  a  high  level  of  process  maturity,  an  organization  needs  professional  process 
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engineers  and  measurement  specialists  who  can  provide  technical  service  support  for  practical 
work.  Employees  with  professional  technical  skills  are  hard  to  keep,  which  is  the  biggest  problem 
our  organization  encountered.  To  maintain  the  stability  of  our  process  engineers  group,  we  have 
to  provide  competitive  salary,  better  welfare,  and  space  for  their  self-development. 

The  growth  of  the  organization  and  training  them  to  meet  high  maturity  expectations.  Rigorous 
Six  Sigma  DESS  and  engineering  process  training  (using  Six  sigma  tools).  Many  waves  of  Black 
Belt  training  and  increasing  awareness  of  DESS  Methodologies  and  tools.  Rigorous  Green  Belt 
training  and  mandatory  GB  certification  for  everyone  in  the  organization.  Organization-wide 
[named  statistical  package]  tool  availability. 

Initial  resistance  from  the  project  managers  during  implementation  of  sub  processes  metrics. 
However,  commitment  from  senior  management,  continuous  coaching  &  mentoring  to  the  project 
team  and  convincing  them  about  the  advantages  of  these  metrics  helped  in  overcoming 
the  initial  resistance. 

1.  There  was  some  schedule  delay  in  selected  projects  due  to  the  effort  spent  on  various  training 
programs.  We  communicated  all  our  clients  about  the  CMMI  initiatives.  Most  of  them  supported 
us  by  accepting  the  schedule  slippage. 

2.  Interpretation  of  the  CMMI  requirements  was  not  clear,  particularly  with  respect  to  usage  of 
high  maturity  tools  like  ANOVA,  Chi  Square,  Monte  Carlo  simulation,  etc.  We  recently  have 
understood  these  requirements  and  started  implementing  them  (as  of  now  more  than  50%  of  the 
projects  are  practicing  these  requirements).  We  started  our  re-assessment  plan  by  the  middle  of 
last  year  (2007)  and  are  planning  to  complete  it  by  the  beginning  of  2009  with  all  these  high  ma¬ 
turity  practices  in  place  across  the  organization. 

1)  We  get  very  few  turn-key  development  projects  (less  than  10%)  from  clients  as  compared  to 
maintenance  projects,  which  results  in  non-usage  of  'development  project’  related  processes 
(software  specifications.  Technical  Solution  etc)  and  metrics  (effort  variance,  schedule  variance 
etc).  To  overcome  this  problem,  we  are  applying  these  processes  and  metrics  to  in-house  devel¬ 
oped  (internal)  projects  and  products. 

Challenges: 

1.  Enabling  across  the  organization  the  usage  of  process  performance  models  and  their  benefits. 

2.  Creating  a  data  culture  across  the  organization. 

How  it  is  addressed: 

1 .  Role- wise  enabling  sessions,  certification  programs,  and  e-Learning  courses 

2.  We  have  created  a  very  strong  data  culture  across  the  organization  through  a  Balanced  Score 
card  approach.  The  business  goals  in  the  score  card  include  process  related  goals  like  quality, 
productivity,  reuse  percentage,  customer  satisfaction  levels  etc.  These  performance  goals  are  re¬ 
viewed  with  board  members  at  regular  frequency  for  mid-course  corrections.  In  most  cases  these 
benefits  are  showcased  to  our  customers  and  have  won  various  accolades. 

There  were  no  barriers  in  the  true  sense.  Eocused  training  and  awareness  helped  in  the  implemen¬ 
tation  of  the  performance  models. 

Understanding  the  expectations  of  Process  Performance  Models  from  SEI  Guidelines  and  then 
developing  the  best  suited  ones  for  our  business  model  was  the  most  difficult  part. 

Our  process  and  process  performance  model  didn't  apply  to  the  fast  extended  business. 
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So  we  subdivided  our  process  and  established  a  new  process  and  process  performance  model  for 
the  extended  business. 

The  overhead  of  CMMI L5  is  high,  which  causes  higher  cost  in  the  projects.  We  cannot  solve  this 
problem. 

The  project  managers  did  not  want  to  use  models  where  the  effects  cannot  be  actually  “felt”  (es¬ 
pecially  through  measurement). 

Education  centered  on  the  purpose  and  the  effect  on  the  organization’s  people.  Explanation  of  the 
new  mind  set  was  repeated  in  quality  audits  every  month. 

Historical  data  was  saved  in  a  distributed  database  which  made  it  difficult  to  analyze. 

We  first  collected  these  data  into  a  centralized  database  with  uniform  data  specifications.  After 
that,  we  performed  further  data  analysis  and  established  process  performance  models. 

Providing  an  effective  Process  Performance  Model  for  the  organization  that  is  useful  and  makes 
sense  to  implement. 

We  failed  to  sufficiently  align  and  prioritize  our  measurement  and  analysis  practices  with  our 
business  and  technical  goals  and  objectives. 

Ereqnent  changes  driven  by  SEI  and  interpretation  challenges  with  CMMI  version  1 .2  have  been  a 
major  barrier  to  achieving  high  maturity. 

To  overcome  these  we  are  monitoring  closely  the  changes  being  driven  by  SEI  and  working 
closely  with  external  consultants  to  adopt  to  the  new  requirements,  apart  from  undergoing  self 
learning  and  in-house  training. 

The  greatest  barriers  are  how  to  verify  the  validity  of  data,  and  we  have  enhanced  training  and 
checking  data. 

The  time  to  market  pressure  for  project  development,  the  deadlines  for  implementation  of  high 
maturity  levels:  Process  tailoring  according  to  the  specific  characteristics  was  used  to  overcome 
these  challenges.  The  value  of  a  process  culture  was  emphasized  by  demonstrating  the  results  of 
the  projects,  statistically  managed  understanding,  specialized  trained  on  statistical  concepts  (proc¬ 
ess  engineering),  theoretical  training  instead  of  practical,  and  executive  involvement. 

The  greatest  barriers  have  been  acquiring  enough  expertise  to  support  the  growing  demands  for 
predictive  models.  We  recognize  the  need  to  understand  our  capability  so  that  as  we  chart  the 
course  in  to  new  market  areas  we  have  a  better  idea  of  what  is  needed  to  be  successful  in  that 
area.  We  continue  to  grow  the  expertise  internally  but  constantly  seek  additional  train¬ 
ing/mentoring  aids  to  augment  our  internal  resources  (people,  training,  tools...). 

People  support  across  all  business  units:  Identification  of  the  right  people,  edncation,  training, 
focus  on  aspect  of  usefulness,  motivation,  awards.  Putting  projects  on  hold  for  various  reasons, 
metrics.  Measurement  data  was  not  sufficient.  Selection  of  alternate  projects  was  time- 
consuming.  People  leaving  the  organization  created  backups  for  everything  and  was  costly,  so  we 
worked  on  retention,  people  policies,  etc. 

Demonstrating  the  value  of  improvements  based  on  quantitative  data. 

We  did  not  have  data  to  make  a  process  performance  model.  Therefore  we  began  with  the  collec¬ 
tion  of  the  correct  data. 

The  business  models  employed  by  our  organization  require  positive  ROI  within  12-24  months  of 
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an  investment.  This  is  often  not  possible  with  higher  orders  of  CMMI  maturity  which  greatly  in¬ 
hibits  investment.  Furthermore  our  customers  rarely  value  the  additional  quality  and  predictability 
sufficiently  to  pay  a  premium  during  the  time  when  the  organization  is  experiencing  the  cultural 
shift. 

Alignment  of  the  entire  organization  in  process  improvement  initiative  (CMMI),  including  facili¬ 
tation,  the  training  program,  project  audits,  project  reviews,  senior  management  review. 

Adapting  the  CMMI  model  to  our  small  project  and  maintenance  environment.  To  overcome  this 
we  made  our  SPC  a  standard  process  that  is  repeatable  rather  than  using  a  lot  of  different  SPC 
techniques  to  look  at  data  in  multiple  different  ways.  This  may  diminish  some  potential  benefits 
from  SPC  but  it  also  allows  us  to  perform  SPC  within  the  budgets  and  time  constraints  for  small 
projects  and  maintenance  projects. 

Selecting  sub-processes 

Use  historical  data  for  statistical  analysis 

Creating  process  performance  models  and  deploying  them  extensively. 

Identifying  process  measures  that  naturally  lend  themselves  to  predicting  meaningful,  actionable 
outcomes  for  project  managers.  Also,  developing  process  performance  baselines  and  models  that 
can  be  useful  to  all  projects  -  operational  factors  like  client  requirements,  SLAs,  etc...  may  intro¬ 
duce  performance  requirements  that  differ  from  our  typical  organization  performance.  We  have 
conducted  user  focus  groups  and  data  analysis  projects  to  understand  modeling  needs  and  to  stan¬ 
dardize  operational  definitions.  We’ve  also  stratified  process  performance  baselines  where  appli¬ 
cable,  and  enabled  project  teams  to  establish  their  own  baseline  limits. 

Finding  resources  to  work  with  PPMs  and  PPBs 

To  get  the  practitioners  to  believe  in  the  benefit  of  using  predictive  models. 

Overcame  this,  to  some  extent,  by  demonstrating  this  with  sample  data. 

Customized  tools  have  been  developed  to  reduce  manual  work  on  data  analysis.  Practitioners 
were  not  convinced  about  the  quality  and  suitability  of  the  tools  initially.  We  had  to  convince  the 
team  through  early  pilot  results  of  the  tools. 

Cultural  barriers. 

I  think  the  greatest  barrier  that  our  organization  faced  is  changing  the  mind  set  of  staff  members 
towards  a  quality  system  and  particularly  metrics  analysis.  We  found  once  the  software  engineer  / 
project  leader  /  project  manager  sees  the  value  in  CMMI  best  practices  the  implementation  be¬ 
comes  smooth. 

It  has  been  a  learning  journey  all  the  way.  Our  systems  and  processes  have  evolved  with  that 
learning.  Change  is  for  the  good,  but  it  takes  time  to  accept.  Over  a  period  of  time  we  have  been 
able  to  institutionalize  the  changes.  And  the  journey  continues  as  we  focus  on  continued  im¬ 
provement. 

Sub  process  implementation  when  the  development  life  cycle  is  short  (e.g.,  the  project  comprises 
a  number  of  reports  which  individually  take  5-20  days  of  effort).  When  the  overall  cycle  time  or 
development  effort  is  small,  implementing  the  sub-process  management  became  challenging.  By 
the  time  measurement  results  are  available,  development  is  complete  without  much  scope  for  con¬ 
trol  at  the  sub-process  level.  The  alternative  in  this  case  is  to  treat  each  of  these  developments  as  a 
sub-process  of  the  over  all  project. 
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Identification  and  implementation  of  prediction  models:  Selection  of  the  right  variables  for  a  pre¬ 
diction  model  is  not  an  easy  task  when  an  organization  is  embarking  on  CMMI L4  implementa¬ 
tion  or  a  CMM  to  CMMI  transition.  Performance  modeling  is  one  area  where  not  many  external 
training  programs  or  consulting  expertise  are  easily  available.  In  most  cases  model  builders  have 
to  rely  on  literature  search  and  data  available  on  the  web  for  designing  a  model  that  would  work 
in  their  own  organizational  environment.  Historical  data  availability  (and  accuracy  of  it  if  avail¬ 
able)  is  the  next  major  challenge.  Many  times  this  leads  to  a  situation  where  models  are  built  us¬ 
ing  whatever  data  is  available.  This  is  one  area  where  the  organization  needs  to  be  prepared  to  do 
trial  and  error  until  you  get  it  right.  The  only  solution  is  to  constantly  watch  for  opportunities  to 
fine  tune  the  model  based  on  the  implementation  results. 

Ensuring  data  accuracy  and  usage:  Many  times  it  will  be  difficult  get  the  right  data  because  of  the 
wrong  usage  of  the  data  (e.g.,  usage  of  data  for  performance  evaluation)  in  the  past.  The  key  chal¬ 
lenge  is  to  restrict  usage  of  the  data  only  for  process  improvement  and  not  for  performance  man¬ 
agement.  Once  people  lose  faith  in  the  data,  it  is  very  difficult  to  ensure  appropriate  usage  of  it  in 
decision  making  and  for  improvement  actions.  A  solution  to  overcome  this  problem  is  de¬ 
coupling  performance  objectives  from  the  metrics  program.  Data  inaccuracy  may  result  from 
wrong  interpretation  of  guidelines,  lack  of  adherence  to  standards  and  /  or  lack  of  awareness  of 
quantitative  management.  Entire  project  teams  need  to  be  trained  on  the  defect  classification 
guidelines,  time  entry  guidelines,  metrics  interpretation  and  statistical  techniques.  Measurement 
errors  can  be  reduced  to  a  great  extent  by  automating  data  collection.  Another  important  means  to 
ensure  metrics  usage  is  to  focus  on  few  metrics  that  are  important  to  the  users’  day  to  day  activi¬ 
ties  rather  than  collect  lot  of  data.  Quality  time  needs  to  be  spent  on  analysis  and  corrective  meas¬ 
ures.  The  results  from  such  actions  should  be  shared  among  potential  users  to  improve  acceptance 
of  quantitative  management  among  them. 

Effect  of  corporate  systems  and  processes  on  a  subsidiary's  high  maturity  journey.  Many  times, 
the  global  processes  and  tools  prescribed  at  the  corporate  level  may  meet  the  CMMI  related  meas¬ 
urement  requirements  or  support  high  maturity  practices.  Corporate  reporting  requirements  man¬ 
date  the  usage  of  such  tools  and  processes  across  the  divisions  /  subsidiaries.  In  such  cases,  sub¬ 
sidiaries  end  up  having  redundant  systems  that  might  add  extra  effort  and  negate  the  benefits  that 
would  have  derived  from  high  maturity  practices. 

•  Collecting  measures  with  needed  frequency,  (action:  direct  involvement  of  quality  team  mem¬ 
bers  with  the  project  teams 

•  Building  the  models  based  on  the  available  data  (action:  dividing  the  measures  into  smaller  units 
based  on  components  or  iteration  or  both  to  have  sufficient  data  to  create  models) 

•  Depending  on  the  output  that  comes  from  the  prediction  models  in  monitoring  and  controlling 
the  project  (action:  extensive  involvement  of  the  quality  team  with  the  PM  for  decision  support 
based  on  prediction  model  outcomes) 

•  Convincing  practitioners  to  use  available  historical  data  in  their  planning  for  the  project  (action: 
extensive  assistance  of  the  quality  team  to  the  PM  during  estimation  and  reviewing  the  estimation 
output  accordingly) 

1 .  Eack  of  consensus  on  common  and  unified  qualified  system  was  a  barrier  when  we  started. 

This  was  overcome  by  establishing  a  SEPG  group  and  having  them  actively  engage  with  the  indi¬ 
vidual  groups  in  a  democratic  approach.  We  eventually  developed  one  of  the  best  processes 
around. 


155  I  CMU/SEI-2008-TR-024 


S7Q3:  What  have  been  the  greatest  harriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them? _ 

2.  Measurement  was  daunting  with  so  many  metrics  to  be  collected  and  so  little  automation,  when 
we  started.  We  overcame  this  by  developing  an  online  metrics  capture  and  tracking  tool.  The  tool 
is  so  much  part  of  our  work  life  today. 

3.  Getting  the  engineer  level  people  to  understand  and  appreciate  the  need  for  a  performance 
model  was  a  big  challenge;  they  would  see  the  process  as  'avoidable'  and  an  obstacle  on  their  way. 
We  overcame  this  by  constantly  having  our  senior  people  establish  and  demonstrate  the  benefits. 
This  enthused  all  others  into  a  participative  approach,  making  quality  the  nature  of  our  work. 


1)  Unavailability  of  people  with  necessary  expertise.  2)  Interpretation  of  L4  &  L5  practices,  espe¬ 
cially  'critical  sub-process'. 

1 )  Obtained  buy-in  from  top  management  to  make  people  available.  2)  Consulted  industry  experts 
&  HMLAs. 

Determining  what,  if  any,  ROI  high  maturity  has  yielded.  There  is  never  enough  time  to  do  things 
that  add  value  to  our  product  in  a  smart  way. 

[Organization  name]  was  assessed  at  CMM  level  5  in  1999  and  later  at  CMMI  level  5  (ver  1.1)  in 
2005.  To  that  extent,  [organization  name]  has  been  an  organization  practicing  high  maturity  for 
many  years.  The  addition  to  the  existing  practices  was  the  introduction  of  the  process  perform¬ 
ance  model  -  the  challenge  faced  was  the  limited  statistical  knowledge  of  the  practitioners.  Exten¬ 
sive  trainings  and  mentoring  were  conducted  for  projects  across  locations  at  various  levels. 

Convincing  technical  project  leaders  that  the  overhead  associated  with  the  data  collection  and 
reporting  is  justified  by  the  benefit.  It's  still  a  work  in  progress. 

Effective  application  of  control  charts  for  development  projects  having  too  few  iterations:  We 
were  not  having  sufficient  data  points  to  plot  control  charts.  We  used  data  from  similar  projects  to 
arrive  at  in  initial  baseline.  We  also  reduced  data  capturing  cycle  time. 

1.  In  a  few  cases  we  have  had  problems  stabilizing  process  performance  data.  We  have  applied  a 
variety  of  techniques  to  overcome  this.  Segmenting  the  data  is  one  of  the  techniques  adopted.  It 
has  yielded  satisfactory  results  in  a  few  or  all  the  segments.  As  a  standard  practice,  we  also  do 
CAR  analysis  on  outliers  to  understand  the  abnormality  and  eliminate  the  causes  for  abnormal 
behavior.  This  has  also  resulted  in  greater  stability  of  data. 

2.  One  of  the  most  important  quality  related  measure  for  us  is  the  defects  leaking  to  the  customer. 
Getting  defect  data  from  the  customer  has  been  a  little  difficult,  given  that  most  of  our  customers 
are  abroad  and  we  have  our  own  onshore  organizations  that  at  times  support  the  user  acceptance 
phase.  To  ensure  quicker  tum-around  time,  sometimes  our  onshore  organizations  or  the  clients 
themselves  fix  the  defects  detected  during  user  acceptance  testing  without  our  knowledge.  Again, 
during  integration  testing,  what  we  develop  needs  to  be  finally  integrated  with  existing  systems  or 
systems  developed  by  other  vendors.  During  this  phase,  quite  often  the  customer  fixes  the  errors 
detected  and  does  not  inform  us.  To  overcome  this,  we  provide  collaborative  tools  like  [named 
vendor]  in  which  everybody  including  the  customer  enters  defects  data  and  such  data  is  available 
to  us  for  our  analysis  and  modeling. 

Collecting  accurate  data 

Project  team  &  managers  are  not  familiar  with  all  types  of  advance  statistical  techniques  used  for 
QPM.  To  overcome  this  issue  the  following  actions  have  been  implemented. 

1 .  The  Metrics  &  EPG  team  had  conducted  various  levels  of  training,  induction  session,  tutorials 
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and  guidelines  to  enhance  the  knowledge  on  SPC  &  PPM. 

2.  Macro-based  spreadsheets  were  designed  to  enable  the  PL/PM  to  implement  advanced  SPC 
techniques  and  PPMs  with  optimal  effort. 

Different  expectations  between  industry  and  SEI  due  to  changing  emphasis  of  what  constitutes 
high  maturity  (apparent  stronger  emphasis  on  PPMs.):  To  better  understand  these  expectations 
and  identify  value  added  opportunities  for  implementation,  we  are  engaging  other  high  maturity 
organizations  to  leverage  their  experiences  as  well  as  collaboration  areas  with  the  SEI. 

Determining  which  measures  will  give  us  the  best  insight  into  our  processes.  Over  time,  we  have 
been  willing  to  eliminate  measures  which  have  been  determined  not  to  add  value  and  introduce 
new  ones  with  potential  to  have  higher  benefit. 

Coordination  and  common  process  among  the  sites  in  our  business  unit.  We  have  a  common 
process  team  and  a  common  measurement  team. 

Sometimes  when  staff  changes,  process  performance  models  may  not  be  well  used  to  predict  pro¬ 
ject  process  performance  since  software  process  performance  is  significantly  influenced  by  em¬ 
ployees. 

No  additional  resources  are  added  in  the  organization  solely  for  the  purpose  of  CMMI. 

All  work  related  to  CMMI  is  integrated  as  one's  job  duty.  CMMI  becomes  the  core  value  of  the 
organization. 

The  greatest  barrier  has  been  the  projects’  inability  to  see  clearly  the  benefits  of  high  maturity. 

We  try  to  overcome  this  barrier  through  training  and  persistent  attempts  to  convince  people  of  the 
benefits  of  high  maturity. 

Onsite  teams  have  a  reduced  level  of  quality  and  process  maturity  and  have  a  reduced  interest  in 
investing  in  bringing  onsite  teams  up  to  same  level  as  offshore. 

Use  of  corporation  baselines  &  models  for  standard  processes  that  are  too  generic  or  high-level  to 
provide  insight  into  standard  process  implementation. 

The  greatest  barrier  faced  was  to  make  people  understand  the  spirit  of  the  process  areas,  rather 
than  blindly  following  them.  We  have  been  able  to  overcome  this  partially,  and  there  are  remark¬ 
able  improvements  in  the  areas  where  the  essence  and  spirit  is  understood.  It  will  require  further 
patience  and  perseverance  to  imbibe  and  institutionalize  it  such  that  it  becomes  part  of  life. 

Changes  in  management  &  differences  in  styles.  Overcoming  the  fear  culture  that  was  in  place 
due  to  previous  management  has  been  a  slow  process  -  building  trust  takes  time,  personality, 
pragmatism,  and  great  communication.  Introducing  culture  changes  quickly  without  impacting  on 
morale  or  the  client  is  always  a  challenge.  It  helps  to  analyze  impact  and  value  of  changes  before 
approval,  making  sure  that  measures  and  'extra'  work  focus  on  business  objectives  and  deliver 
visible  value. 

One  of  the  major  barriers  we  faced  early  in  our  quality  journey  was  changing  mindsets  of  indi¬ 
viduals  as  we  moved  towards  implementing  organization  project  management  practices  based  on 
statistical  thinking.  This  was  overcome  largely  due  to  continued  commitment  and  support  from 
senior  management  towards  process  improvement  initiatives.  We  also  under  took  some  formal 
organization  change  management  programs. 

Another  area  where  we  faced  challenges  was  adequate  training  for  project  managers  and  staff  on 
quantitative  methods  and  statistical  techniques.  We  overcame  this  by  increasing  budget  allocation 
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S7Q3:  What  have  been  the  greatest  harriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them? _ 

for  various  training  efforts. 

1)  The  availability  of  historic  and  reliable  data  was  a  challenge  when  organizations  were  in  opera¬ 
tion  for  only  few  years  at  the  time  of  our  CMMI  implementation  journey.  That  was  especially  true 
with  limited  but  large  projects  (insurance  software  implementations)  when  embarking  on  a  jour¬ 
ney  for  a  high  maturity  level.  We  tried  sampling,  using  industry  benchmarks  for  known  areas,  and 
also  waited  to  implement  M&A  and  HML  practices  until  we  could  accumulate  statistically  sig¬ 
nificant  data. 

2)  The  perception  of  people  that  too  much  measurement  and  analysis,  data  collection  etc  con¬ 
sumes  a  lot  of  time;  We  challenged  this  perception  by  sharing  the  benefits  that  these  same  proc¬ 
esses  have  provided  to  the  projects,  the  organization  and  also  how  they  have  helped  in  improving 
the  quality  of  work  life.  Now  people  see  processes  as  something  that  will  help  them  rather  then  a 
burden  for  the  sake  of  company. 

3)  When  we  were  undergoing  an  implementation  of  CMMI  high  maturity,  staff  had  enthusiasm 
and  drive  towards  achieving  the  maturity  level.  However,  to  keep  them  equally  motivated  after 
achievement  was  a  big  challenge.  We  have  overcome  this  through 

•  Extensive  checkpoints/  gates  and  audits  at  every  process 

•  Mistake  proofing  all  the  key  areas  and  ensuring  refresher  training  at  periodic  intervals 

•  Communicating  the  achievements  through  processes 

•  Adoption  of  a  carrot  and  stick  approach  for  process  compliance  &  non-compliance  respectively 

•  Continuous  alignment  of  processes  (and  improving  them)  to  meet  organization  objectives 

Journey  was  difficult  due  to  unavailability  of  relevant  info  regarding  the  "application  of  CMMI  in 
the  embedded  product  development  environment." 

In  the  beginning  the  lack  of  standardization  for  data  collection.  To  correct  this  we  defined  stan¬ 
dards  as  well  as  criteria  of  good  capture. 

Lack  of  culture  to  collect  some  data,  for  example  functional  points,  fails.  To  correct  this  we  gen¬ 
erated  a  strategy  to  reinforce  the  knowledge  and  the  culture  to  collect  data  as  well  as  the  inclusion 
of  standards  and  the  modification  of  the  tools  that  facilitate  their  registry. 

The  idea  that  quantitative  analysis  represents  more  work,  because  it  implies  more  reports.  To  de¬ 
crease  this  we  developed  some  automatic  tools  that  give  the  projects  the  information  on  time  and 
in  the  form  that  is  required  for  quantitative  analyses. 

In  the  beginning  we  intended  to  quantitatively  manage  a  lot  of  indicators  and  all  process  in  all  the 
technologies  and  services.  Since  then,  we  have  identified  and  prioritized  the  indicators  and  sub 
process  that  would  be  quantitatively  managed  considered  the  impact  in  our  vision  and  mission. 

In  the  beginning  our  data  point  size  was  a  project.  We  changed  it  and  now  we  use  a  small  unit  that 
we  call  a  work  package  and  sub  process. 

The  improvements  to  common  causes  were  defined  by  a  group  outside  the  operation  teams,  and  it 
was  very  difficult  to  deploy  those  improvements.  We  now  have  created  improvement  teams  with 
personnel  from  the  operation  for  common  causes  analysis  and  the  optimization  of  process  per¬ 
formance. 

Tailoring  processes  commensurate  to  business  risk,  customer  process  maturity  diversity,  respond¬ 
ing  with  new/  refined  processes  for  new  business  models  in  a  timely  manner,  deploying  processes 
systematically  and  quickly  to  a  growing  globally  diverse  workforce:  All  of  these  are  systemati¬ 
cally  addressed  through  an  improvement  program  executed  collaboratively  with  stakeholders. 
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S7Q3:  What  have  been  the  greatest  harriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them? 

Assimilation  of  data  and  varied  nature  of  service  contracts;  We  did  introduce  EPM  to  streamline 
project  operational  data. 

The  largest  barrier  is  the  culture  shift,  people  accepting  the  processes  as  the  integral  part  of  their 
practice.  We  work  on  overcoming  it  by: 

-  developing  processes  that  enhance  the  best  practice  of  the  organization 

-  involving  practitioners  in  defining  processes 

-  having  dedicated  resources  in  SEPG  and  QMG  (full  time  assignment) 

-  maintaining  constant  presence  in  projects  (consulting  on  process  tailoring  and  implementation, 
preparing  data  for  analysis,  helping  with  the  analysis  and  decision  making) 

-  communicating  frequently  -newsletters,  lunch  &  learn  sessions,  formal  training  sessions  (inter¬ 
nal  training),  presentations 

-  having  strong  support  from  management  team 

How  to  build  performance  models  fit  to  the  organization 

Eevel  3  measurements  and  analysis  as  the  end  state  of  M&A  has  been  a  potential  limiting  barrier. 
We  move  past  this  by  emphasizing  that  we  need  to  achieve  predictable  performance  and  capabil¬ 
ity  ranges  for  critical  processes  that  meet  specified  goals  and  objectives.  This  will  enable  us  to  be 
able  (eventually)  to  assess  which  of  these  processes  are  influencing  program  performance  (e.g., 
positively)  and  by  how  much,  and  determine  if  the  voice  of  the  business/customers  expectations 
are  being  met  or  we  need  to  refocus  on  other  processes  that  will  be  measurably  more  influential 
for  a  program’s  performance. 

Use  of  statistical  techniques  and  interpretation  of  statistical  data  was  an  area  of  concern  for  us  as 
the  practitioners  (project  groups)  were  not  uniformly  conversant  with  the  same.  This  affected  the 
appropriateness  and  quality  of  corrective  actions  at  the  project  level. 

Several  training  sessions  by  external  experts  helped  us  substantially  to  get  over  this  problem  and 
substantially  improve  effective  use  of  the  data  and  metrics. 

The  greatest  barriers  have  been  resistance  to  doing  extra  work.  Work  to  show  the  teams/managers 
that  it  is  not  extra  work;  it’s  what  they  are  already  doing.  They  just  need  to  use  the  data  and  im¬ 
prove. 

It's  difficult  to  pick  up  critical  factors  while  we  do  process  performance  modeling.  Developers’ 
technical  experience  always  become  the  critical  factor. 

The  senior  staff  will  not  put  enough  emphasis  on  it,  and  they  care  about  only  short-term  things. 
And  because  of  this,  project  managers  will  also  focus  on  fixing  current  work  and  current  status 
only. 

1.  To  understand  the  requirements  of  the  CMMI  model  and  the  "deep  reasons"  why  those  re¬ 
quirements  were  selected.  This  was  overcome  by  using  extensive  consulting  help  and  training  (in- 
house  and  contracted). 

2.  Historical  data  availability.  Since  the  organization  is  characterized  by  a  small  number  of  large 
projects  -  it  was  very  difficult  to  find  consistent  and  complete  information  across  years  (before  the 
CMMI  implementation  effort  started).  A  great  deal  of  labor  was  invested  in  locating  and  validat¬ 
ing  the  consistency  of  "private"  databases  (and  we  were  pretty  lucky  to  find  historical  data...) 

3.  Implementation  of  QPM  across  all  projects.  It  required  (and  is  still  requiring)  high  level  man¬ 
agement  attention  for  doing  the  QPM  practices  in  a  routine  manner. 
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S7Q3:  What  have  been  the  greatest  harriers  faced  by  your  organization  during  its  journey 
to  high  maturity?  What  have  you  done  to  overcome  them? 

There  are  some  factors  that  are  not  under  statistical  management  hut  do  influence  the  outputs  of 
our  modules,  e.g.,  the  customers'  processes.  People  skills,  experience,  etc.  also  need  to  be  treated 
as  PPM  factors,  but  that  still  is  in  progress.  We  try  to  discuss  those  things  with  customers,  and 
have  reached  some  agreement. 

Programs  do  not  always  see  an  immediate  ROI  for  collecting  and  analyzing  measures.  Besides 
making  measurement  collection  a  requirement,  the  quality  organization  participates  in  the  collec¬ 
tion  and  analysis  of  metrics.  This  service  has  proved  beneficial. 

Convincing  practitioners  about  the  usefulness  of  Process  Performance  Models  and  getting  their 
support  in  measuring  the  necessary  parameters  was  a  challenge.  This  happened  in  spite  of  man¬ 
agement  commitment.  During  daily  project  management  activities,  use  of  Process  Performance 
Models  are  not  taken  as  a  priority  at  times  or  put  to  correct  use. 

Gaining  sufficient  resources  throughout  the  organization  to  support  an  advanced  measurement 
program,  including  generation  and  use  of  process  performance  models  to  drive  process  and  tech¬ 
nology  change.  Also,  understanding  /  interpreting  the  CMMI. 

1 .  A  perception  that  Maturity  Models  can  be  adopted  only  by  big  players  existed  among  the 
members  of  the  delivery  organization.  This  perception  had  to  be  removed  through  education  and 
sharing  of  case  studies  from  small  and  medium  players  who  have  adopted  maturity  models  suc¬ 
cessfully. 

2.  Again,  a  perception  that  the  high  cost  of  adopting  a  maturity  model  overweighs  the  benefits 
derived  from  it.  Highlighting  the  importance  of  the  intangible  benefits  of  predictable  quality  has 
helped  us  combat  this  perception. 

3.  Lack  of  clear  knowledge  of  the  benefits  of  process  modeling  leading  to  lack  of  involvement  at 
the  grassroots  level.  Constant  education  and  discussion  in  the  quality  council  meetings  has  helped 
us  reinforce  knowledge  and  its  importance  to  the  target  audience. 

4.  Resistance  to  change! 


In  Conclusion:  Is  there  anything  else  that  you  would  like  to  tell  us  about  the  use  or  useful- 
ness  of  measurement  and  analysis  in  your  organization? 

Measurement  program  has  helped  us  to  realize  the  improvements  quantitatively  and  also  identify 
new  areas  for  improvement. 

Measurement  and  Analysis  are  central  to  our  organization's  ability  to  understand  the  current  state 
and  predict  the  future  state  of  projects.  It  is  inherent  in  how  we  design  our  systems. 

We  would  like  to  refer  to  effective  cases  in  other  organizations  to  improve  our  process. 

You  can  only  manage  what  you  can  measure,  and  understanding  variation  is  the  key  to  managing 
chaos. 

The  measurement  and  analysis  processes  have  been  defined  and  implemented  in  the  organization. 
But  there  is  a  continuous  drive  required  for  ensuring  that  the  organization  baseline  and  models  are 
used.  At  the  time  of  usage  and  seeking  concessions  every  project  appears  to  be  an  exception. 

Metrics  and  measurements  have  become  our  organization  DNA  and  helped  us  to  achieve  greater 
success. 
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In  Conclusion:  Is  there  anything  else  that  you  would  like  to  tell  us  about  the  use  or  useful- 
ness  of  measurement  and  analysis  in  your  organization? 

For  customer-oriented  purposes,  our  measurement  and  analysis  needs  customizing  to  adapt  with 
each  customer  requirement.  Thus,  a  number  of  sub-systems  of  measurement  and  analysis  need  to 
be  set  up  and  maintained. 

Given  some  new  guidance  from  the  SEI  we  now  have  a  better  understanding  of  how  to  develop 
and  implement  new  models  and  expect  our  usage  to  improve  over  the  next  year. 

We  have  come  a  long  way.  There  are  still  pockets  of  resistance  but  overall  measurement  and 
analysis  is  becoming  institutionalized. 

Complexities  in  the  project  governance  models  complicate  the  use  of  the  traditional  approach  of 
measurement  and  analysis,  especially  when  projects  have  different  customer  objectives,  different 
services  executed  within  the  same  project  and  also  requirements  changing  from  the  customer  side. 
Getting  customer  satisfaction  becomes  the  primary  objective  and  inputs  for  that  go  beyond  pure 
process  performance  and  include  certain  intangibles  such  as  attitude,  proactiveness,  resource 
skills  and  stability,  etc. 

We  have  institutionalized  a  metrics-based  management  culture  in  the  organization.  Executive 
management  is  always  informed  about  process  effectiveness.  The  data  task  level  collected  is 
rolled  up  to  get  useful  insights  at  the  project,  account,  business  vertical  and  organization  levels. 
Decisions  on  improving  organization  process  performance  are  taken  based  on  measurement  and 
analysis  which  are  reported  in  the  form  of  management  reports  and  dashboards. 

Measurement  analysis  activities  have  helped  us  align  measurements  activities  with  the  business 
goals,  and  provide  a  quantitative  understanding  of  both  the  process  and  the  product  that  is  built. 
Analysis  also  identifies  areas  of  improvements  and  situations  that  need  corrective  action  to  ensure 
achievement  of  objectives. 

1.  The  most  useful  things  about  [organization’s  measurement  and  analysis  process]  is  the  control 
charts  that  help  us  manage  our  projects  better. 

2.  It  also  is  a  major  process  that  helps  us  better  estimate  and  manage  the  projects. 

The  introduction  of  measurement  and  analysis  practices  and  models  has  been  a  great  experience 
for  us. 

Very  important  also  for  customer  and  staff  satisfaction,  as  well  as  the  company’s  overall  success. 

Because  of  the  High  Maturity  journey,  the  organization  has  been  able  to  instill  "statistical  think¬ 
ing"  in  its  DNA.  Decisions  in  projects  are  based  on  past  performance  and  analysis  of  the  perform¬ 
ance  modeling.  Processes  are  composed  using  models  and  critical  subprocesses  are  controlled 
using  control  charts.  Effectiveness  of  causal  analyses  is  evaluated  statistically  thereby  eliminating 
ambiguity.  Improvement  Initiatives  are  evaluated,  piloted  and  deployed.  Benefits  realized  are 
validated  statistically. 

To  summarize,  [organization]  encourages  data  based  decision  making. 

As  mentioned  above,  [named  quality  measurement]  has  been  the  mantra  in  the  organization  for 
many  years.  This  is  tracked  every  month  at  the  various  levels  of  management  reviews  and  deci¬ 
sions  are  taken  based  on  the  analysis  of  the  data  that  helps  us  to  improve  our  process  perform¬ 
ance. 
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In  Conclusion:  Is  there  anything  else  that  you  would  like  to  tell  us  about  the  use  or  useful- 
ness  of  measurement  and  analysis  in  your  organization? _ 

Useful  in 

1.  Managing  projects  more  proactively  by  taking  mid-term  corrective  actions 

2.  Providing  mechanism  to  foresee  problems  at  an  early  stage  of  the  [specified  life  cycle] 

3.  Identifying  granular  &  controllable  sub  process  parameters,  effective 
in  managing  projects  better 

Measurement  and  Analysis  in  particular,  and  the  implementation  of  various  Quality  Processes 
have  resulted  in  the  reduction  of  costs  through  the  hiring  of  a  very  young  work  force  that  delivers 
the  same  level  of  quality  that  can  be  expected  from  a  more  mature,  more  expensive  work  force. 
Essentially  we  have  moved  from  software  development  as  an  individual  capability  to  software 
development  as  a  group  capability  wherein  different  members  of  a  team  focus  on  individual  as¬ 
pects  of  a  major  software  application  and  collaborate  with  each  other  to  provide  knowledge  as 
needed  by  other  members  of  the  team.  Our  effort  estimation  variance  has  decreased  from  over 
25%  in  December  2003  to  just  over  3.4%  by  March  2008. 

All  our  efforts  in  quality  processes  are  based  on  providing  "Delight"  to  our  customers.  We  do  our 
customer  satisfaction  surveys  through  [named  vendor].  As  per  [named  vendor]  the  customer  satis¬ 
faction  benchmark  is  7.5  on  a  scale  of  10  whereas  we  have  consistently  achieved  10%  higher  rat¬ 
ings  than  this  industry  average. 


1)  Automation  of  data  capturing,  data  analysis  and  reporting:  Our  project  management  tool  pro¬ 
vides  online  efforts  and  schedule  variance  at  phase,  ticket,  feature  (module)  and  project  levels. 

2)  [Named  tool]  has  a  significant  feature  of  publishing  and  releasing  data  through  dashboards  at 
both  the  organization  and  project  level.  This  helps  in  finding  the  process  health  online 

3)  [Named  tool]  provides  many  features  such  as  (a)  resource  productivity  and  utilization,  (b)  de¬ 
fects  related  reports,  (c)  measurements  and  variance  related  reports,  (d)  other  reports  such  as 
timesheet  status,  project  status,  resource  status  etc. 

4)  [Named  tool]  also  helps  in  capturing  data  on  non-traditional  areas  such  as  responsiveness  of 
systems  and  our  networking  group  (problem  tickets  in  systems  and  hardware),  human  resource 
related  issues,  general  admin  (logistics  related  problems). 

Though  measurement  and  analysis  does  not  directly  demand  the  usage  of  sophisticated  tools  such 
as  [named  statistical  package],  we  need  tools  of  this  sort  to  enable  better  and  easy  use  of  the  data 
and  better  interpretation  of  the  data  sets  that  are  available  to  us.  Measurement  and  analysis  is 
helping  the  organization  take  informed  decision  about  its  projects  which  in  turn  helps  the  organi¬ 
zation  make  strategic  decisions  based  on  its  quantitative  management  reports. 

Please  teach  me  useful  measurements  item  for  the  development  of  software. 

Extremely  useful  but  unfortunately  interpretation  guidelines  not  provided  earlier  on  for  users. 

We  would  like  to  share  some  experiences  of  other  organizations'  measurement  and  analysis. 

It  takes  time  to  achieve  high  maturity;  there  is  little  information  about  modeling  and  baselines, 
according  to  the  budget  of  median  companies.  It  is  necessary  dedicate  specialized  personnel  to 
create  the  models  and  baselines.  And  it  is  necessary  to  provide  direct  access  to  SEI  specialists  to 
support  these  concepts.  Benchmarks  of  similar  industries  are  not  available  for  statistically  man¬ 
aged  data  for  developing  better  process  performance  baselines  (PPB)  and  models  (PPM). 
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In  Conclusion:  Is  there  anything  else  that  you  would  like  to  tell  us  about  the  use  or  useful¬ 
ness  of  measurement  and  analysis  in  your  organization? 

Senior  management  recognizes  the  importance  of  understanding  our  performance  capabilities 
through  measurement  and  analysis.  They  continue  to  challenge  the  organization  toward  develop¬ 
ing  an  end  to  end  approach  to  measurement  and  analysis  in  each  phase  of  product  development. 
Our  primary  focus  has  been  in  Engineering  but  the  push/trend  to  understand  the  performance  at 
inception/proposal  through  project  closeout  is  ever  increasing. 

Measurements  and  analysis  is  any  time  and  every  time  useful  to  my  organization. 

Use  of  an  appropriate  tool  for  data  collection  is  key  to  MA  success,  as  is  communication  of  the 
outcome  of  resulting  SPC  to  the  wider  organization  so  practitioners  are  aware  the  data  is  being 
used  in  the  interests  of  continuing  improvement. 

Helped  us  define  and  analyze  the  important  metrics  to  collect 

We  are  using  Six  Sigma  projects  for  improving  our  processes.  Six  Sigma  requires  a  measurement 
system,  analysis  and  high  quality  data.  We  have  found  that  the  metrics  data  available  in  our  data¬ 
base  is  of  great  value  for  these  projects. 

Measurements  are  the  key  to  evaluate  how  the  organization  is  performing.  The  metrics  defined  at 
the  organization  level  are  aligned  to  our  business  goals  and  customer  satisfaction.  The  same  con¬ 
cept  flows  down  to  the  project  level.  The  emphasis  on  quantitative  management  has  resulted  in 
our  project  managers  looking  ahead  and  taking  course  corrections  earlier,  rather  than  being  reac¬ 
tive. 

Once  we  established  the  measurement  systems,  we  could  benchmark  ourselves  with  the  best  in 
the  industry.  This  was  like  reality-check  and  correction.  This  helped  us  improve  our  project  suc¬ 
cess  ratios  suhstantially. 

Measurement  and  Analysis  also  gives  a  feeling  of  pride  to  our  associates.  They  know  that  the  way 
we  do  our  software  engineering  is  quite  different  from  what  they  have  experienced  earlier. 

Better  insight  &  quantitative  monitoring  of  project  progress,  ability  to  measure  product  &  process 
quality,  adoption  of  a  data-driven  decision  making  approach 

We  do  M&A  very  well. 

The  process  performance  model  gives  the  projects  a  clear  view  of  the  current  and  future  product 
and  process  quality.  It  thus  enables  them  to  take  corrective  actions  guided  by  the  what-if  analysis. 

It’s  widely  used  to  predict  schedule  and  effort  variance  based  on  the  project  health  sheet. 

The  leading  indicators  help  some  projects  in  re-planning  and  critical  decision  making.  This  is  now 
being  propagated  across  the  organization. 
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General  observation: 

1 .  Performance  models  on  size,  effort,  defect  and  schedule  have  maximum  acceptance 

2.  Measures  need  to  relate  to  organizational  goals 

3.  Project  manager/leaders  education  on  measurement  &  analysis  is  key  to  success. 

Success: 

1 .  We  have  done  pioneering  work  in  the  measurement  of  size  and  defining  productivity  bench¬ 
marks  in  package  products  like  [named  proprietary  products].  It  has  improved  our  estimation  and 
planning  processes  substantially. 

2.  We  have  also  done  original  work  in  collecting,  analyzing,  segmenting,  modeling  and  bench¬ 
marking  productivity  data  in  the  form  of  ticket  resolution  effort  and  time  in  the  areas  of  applica¬ 
tion  and  infrastructure  management.  Deployment  of  high  maturity  principles  here  has  been  highly 
beneficial  to  us 

3.  By  combining  Six  Sigma  and  Lean  techniques  combined  with  high  maturity  principles,  we 
have  experienced  improvements  in  productivity,  cycle  time  and  quality  in  the  range  of  10-25%  at 
the  organization  level  and  between  10-65%  at  individual  project  levels. 

We  feel  we  have  a  strong,  goal-based  measurement  infrastructure  that  supports  a  robust  business 
and  program  rhythm  of  review  to  provide  insights  into  performance,  early  corrective  actions,  and 
performance  improvement  opportunities.  Tools  are  in  place  to  automate  data  collection,  program 
and  organizational  reporting  and  analysis.  Strong  top  down  management  support  exists  for  the 
measurement  and  analysis  program. 

As  a  fairly  recent  high  level  maturity  appraised  organization,  our  ability  to  benefit  from  our 
measurement  and  analysis  program  is  improving  as  our  database  grows  and  as  we  experience 
where  we  get  the  most  bang  for  the  buck.  We  also  have  plans  to  increase  the  sophistication  of  our 
tools  over  the  coming  year. 

Our  model  builders,  maintainers,  mentors  and  coaches  are  all  part  of  our  technical  project  staff. 
Although  measurement  and  analysis  is  not  their  primary  assignment,  they  do  have  a  percentage  of 
their  time  dedicated  to  their  roles  in  our  measurement  program. 

We  are  increasing  the  effort  around  our  quality  systems  and  measures. 

Application  of  M&A  and  other  CMMI  practices  has  definitely  helped  us  in  achieving  our  set  ob¬ 
jectives  to  a  great  extent.  Today,  we  can  confidently  say  with  supporting  data  that  we  have  been 
able  to  improve  in  many  areas.  We  stand  on  a  better  platform  altogether  from  when  we  started  a 
few  years  back. 

One  of  the  recommendations  that  I  would  like  to  make  to  SEI  is  to  include  in  the  standard  “over¬ 
view  of  CMMI”  training  sessions  to  not  only  mention  “what  an  organizations  should  do”  to  im¬ 
plement  high  maturity  levels  but  also  “how  to  do”  it  with  details  on  commonly  used  tools,  hands- 
on  and  case  studies  for  the  participants.  Participants  for  such  training  usually  end  up  just  knowing 
the  standard  with  very  little  clue  on  the  tools,  their  implementation  and  most  importantly  what 
kinds  of  value  they  can  derive  from  high  maturity  practices  particularly  that  are  related  to  process 
capabilities  including  PPMs.  The  training  does  convey  the  value  but  it  is  mostly  theoretical. 

It  is  only  when  a  hired  consultant  educates  the  participants/staff  of  the  organization  on  their 
CMMI  HML  journey  that  they  will  get  the  deeper  insight  into  HML  practices.  I  have  personally 
experienced  as  an  ATM  during  the  assessment  of  maturity  level  5  that  sometimes  an  interpreta¬ 
tion  of  the  hired  consultant,  of  a  particular  practice  and  the  documents  that  can  support  it,  is  dif¬ 
ferent  from  the  interpretation  of  the  assessors  /  Lead  appraisers  which  in  turn  is  negative  for  the 
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organization.  If  SEI  can  have  a  database  on  common  misinterpretations  it  will  help  organizations 
in  implementation  HML  practices. 

The  major  benefits  that  we  identify  are: 

Having  a  better  understanding  about  the  quality  of  the  product  allows  better  management  deci¬ 
sions  with  highly  satisfied  customers. 

Process  Improvement  implementations  are  managed  by  quantitatively  based  decisions  and  aligned 
to  the  organization  vision  and  mission. 

Measuring  the  cost  of  finding  and  fixing  defects. 

We  now  can  make  project  to  project  comparisons  in  order  to  understand  their  variation. 

Common  processes/measures  allow  better  use  of  historical  data. 

Calibrating  cost  estimation  models  has  led  to  better  estimation. 

Align  measurement  and  analysis  tightly  with  overall  business  objectives  and  as  a  part  of  overall 
business  excellence.  The  measurement  and  statistical  analysis  should  be  simple  and  easy  to  un¬ 
derstand. 

Adopting  this  model  has  forced  us  to  customize  the  modeling  to  our  needs  and  set  benchmarks 
and  try  to  outperform  the  same. 

It  would  be  useful  to  have  webinars  organized  to  share  industry  challenges  and  various  user  feed¬ 
back  on  overcoming  them  across  different  process  areas.  There  is  not  much  data  available  on  soft¬ 
ware  service  industry  (non  products)  models. 

Though  the  initial  impression  was  of  added  effort  towards  data  collection  and  interpretation  at  the 
project  group  levels,  proper  training,  mentoring  and  performance  improvement  have  changed  the 
mind-set  of  almost  everyone,  and  the  activity  has  now  been  well-accepted  as  an  integral  part  of 
tbe  work. 

1.  Regarding  selection  of  subprocesses  base  on  their  capabilities.  We  are  using  standard  proc¬ 
esses/subprocesses  where  tailoring  is  basically  done  only  upon  contractual  requirements.  We  did¬ 
n't  come  to  a  situation  where  we  need  to  choose  a  subprocess  (out  of  set  of  alternative  subproc¬ 
esses)  based  on  each  subprocess  capability. 

2.  It  is  hard  to  complete  a  CAR  process  change  within  the  project  where  a  problem  was  detected, 
since  the  detection,  process  change,  piloting  the  new  process  with  statistical  data  (not  to  mention 
deployment  in  other  projects)  cycle  is  in  many  cases  longer  than  the  time  frame  of  the  project 
where  the  process  problem  was  detected. 

To  conclude,  we  gain  a  lot  from  high  maturity  implementation.  Although  it  is  difficult  to  calculate 
an  ROI,  we  have  seen  improvements  on  all  performance  indicators  at  the  organization  level! 

Measurement  and  Analysis  should  be  used  in  the  context  wherein  it  aids  the  organization  to 
achieve  its  business  objectives.  In  many  occasions,  organizations  could  be  using  a  hammer  to  kill 
a  fly  by  deploying  an  overweight  M&A  system  complying  with  QPM  and  OPP  requirements  of 
CMMI.  Add  to  this,  the  GP  2.8  requirements!  Judicious  usage  of  M&A  practices  and  meaningful 
application  of  CMMI  PAs  are  the  keys  to  success  in  getting  the  management  and  practitioners' 
bny  in.  That  should  be  the  focus  in  the  appraisals  too! 

The  use  of  measurement  and  analysis  has  enabled  us  and  our  clients  to  "feel"  tbe  quality  as  op¬ 
posed  to  evaluating  quality  only  from  a  set  of  numbers.  As  the  only  [named  domain]  company  to 
have  gone  for  maturity  level  4,  we  are  happy  that  we  have  gotten  there  first.  Now  that  we  have 
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reached  a  state  of  consolidation,  we  are  inspired  to  get  to  the  next  level  soon. 

Nothing,  thank  you.  { 12  other  respondents  gave  similar  answers.} 

[Company  name]  is  a  growing  organization;  certainly  most  of  these  practices  are  very  useful  for 
us.  SEI  can  bring  out  guidelines  for  the  implementation  of  CMMI  in  small  projects  (specific  to 
high  maturity  practices). 

[Named  organization  provided  a  two  page  summary  of  its  business  history  and  scope,  along  with 
a  description  of  its  measurement  program  and  process  performance  modeling  work.] 

We  have  consistently  grown  at  the  rate  25%  to  30%  per  year  and  still  we  could  achieve  the  fol¬ 
lowing  among  other  things  for  our  customers: 

1.  Repeat  business  of  close  to  95%  with  higher  engagement  level  ratings. 

2.  We  have  seen  productivity  improvement  of  7%  to  10%  per  year  with  world  class  quality.  Close 
to  80%  of  projects  deliver  zero  defects 

3.  Consistent  delivery  on  time.  97%  of  projects  deliver  on  or  before  agreed  timelines  in  applica¬ 
tion  development  and  maintenance  projects 

We  have  well-integrated  measurement  tools  ([named  COTS  and  proprietary  tools])  for  effort, 
size,  schedule  and  defect  tracking  which  help  in  collecting  the  metrics  at  the  point  of  action.  The 
tools  also  have  mechanisms  to  ensure  integrity  and  completeness  of  data.  We  have  dedicated  peo¬ 
ple  on  the  metrics  team  who  are  responsible  for  aggregating  and  analyzing  the  project  level  data 
and  building  Process  Performance  Baselines  at  the  organization  level.  The  PPBs  are  usually  re¬ 
vised  every  quarter.  This  data  is  used  to 

1 .  Benchmark  our  performance  with  the  best  in  class  companies  and  identify  improvement  oppor¬ 
tunities 

2.  Provide  historical  data  to  quantitatively  predict  the  performance  of  the  projects 

3.  Monitor  the  performance  at  the  organization  and  project  level  to  take  proactive  steps 

We  have  also  created  our  own  Process  performance  models  for  schedule  optimization  based  on 
historical  data  that  are  used  to  predict  schedule  and  performance.  Our  [named]  health  dashboard 
uses  leading  indicators  to  predict  future  performance.  Our  automated  milestone  reports  help  moni¬ 
tor  the  performance  in  projects/accounts/units  quantitatively. 

The  measurement  program  is  very  unique  because  it  sets  the  direction  for  many  organization  level 
improvement  initiatives,  score  cards  for  senior  management,  etc.  The  data  is  translated  to  measur¬ 
able  and  manageable  goals  at  all  levels  up  to  project  manager.  The  scorecard  performance  is 
measured  every  quarter.  A  portion  of  salary  is  linked  to  meeting  the  scorecard  goals. 

This  survey  appears  to  be  aimed  at  creating  products  and  services  to  help  organizations  move  be¬ 
yond  CMMI  level  3.  No  such  services  are  currently  being  sought  by  this  organization. 
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I  would  like  to  see  SPC  become  more  useful  to  smaller  organizations  wishing  to  obtain  benefits 
from  these  process  areas.  Too  many  of  the  requirements  for  higher  maturity  level  process  areas 
seem  to  be  stuck  in  the  academic  world  rather  than  creating  good  real  world  practices  that  can 
scale  level  4/5  process  areas  for  smaller  organizations.  QPM,  OPP,  CAR  and  OID  should  be  able 
to  be  done  in  repeatable  processes  that  work  for  1-3  person  teams  (rather  than  just  10-100  person 
teams).  Our  organization  had  successfully  implemented  several  streamlined  level  4/5  processes, 
bnt  then  a  few  years  later  the  SEI  determined  that  they  would  turn  up  the  gain  on  higher  maturity 
levels  and  put  them  out  of  reach  (from  a  business  perspective)  of  smaller  organization.  I  think  the 
SEI  should  rethink  how  level  4/5  process  areas  must  be  met  so  as  to  enable  smaller  organizations 
to  participate  in  these  valuable  process  areas. 
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